IEA Research for Education 
A Series of In-depth Analyses Based on Data of the International 
Association for the Evaluation of Educational Achievement (IEA) 


r a Lh ATE ^ "m 
a AR e Wi 
dn 


lv c mp (Nee 
72 VW ^ Ee & ^ 


Barbara Japelj Pavešić 
Paulina Korsnakova 
Sabine Meinck Editors 


Dinaric Perspectives 
on TIMSS 2019 


Teaching and Learning Mathematics 
and Science in South-Eastern Europe 


m 
E. hk 
(S 
Se 


2) Springer 


IEA Research for Education 


A Series of In-depth Analyses Based on Data 
of the International Association for the Evaluation 
of Educational Achievement (IEA) 


Volume 13 


Series Editors 
Seamus Hegarty, University of Warwick, UK, University College Dublin, Ireland 
Leslie Rutkowski, Indiana University, Bloomington, USA 


Editorial Board 
John Ainley, Australian Council for Educational Research, Australia 
Sarah Howie, Stellenbosch University, South Africa 


Eckhard Klieme, German Institute for International Educational Research (DIPF), 
Germany 


Rainer Lehmann, Humboldt University of Berlin, Germany 
Dominique Lafontaine, University of Liége, Belgium 
Fou-Lai Lin, National Taiwan Normal University, Taiwan 
Marlaine Lockheed, Princeton University, USA 

Sarah Maughan, AlphaPlus Consultancy, UK 


Maia Miminoshvili, President, Education Policy and Research Association 
(EPRA), Georgia 


Carina Omoeva, FHI 360, USA 

Elena Papanastasiou, University of Nicosia, Cyprus 
Valena White Plisko, Independent Consultant, USA 
David Rutkowski, Indiana University, USA 

Franck Salles, Ministère de l'Education nationale, France 
Andres Sandoval Hernandez, University of Bath, UK 
Jouni Valijarvi, University of Jyvaskyla, Finland 


Hans Wagemaker, Senior Advisor to IEA, New Zealand 


The International Association for the Evaluation of Educational Achievement 
(IEA) is an independent nongovernmental nonprofit cooperative of national 
research institutions and governmental research agencies that originated in 
Hamburg, Germany in 1958. For over 60 years, IEA has developed and conducted 
high-quality, large-scale comparative studies in education to support countries’ 
efforts to engage in national strategies for educational monitoring and improvement. 

IEA continues to promote capacity building and knowledge sharing to foster 
innovation and quality in education, proudly uniting more than 60 member 
institutions, with studies conducted in more than 100 countries worldwide. 

IEA’s comprehensive data provide an unparalleled longitudinal resource for 
researchers, and this series of in-depth peer-reviewed thematic reports can be used 
to shed light on critical questions concerning educational policies and educational 
research. The goal is to encourage international dialogue focusing on policy matters 
and technical evaluation procedures. The resulting debate integrates powerful 
conceptual frameworks, comprehensive datasets and rigorous analysis, thus 
enhancing understanding of diverse education systems worldwide. 


More information about this series at https://link.springer.com/bookseries/14293 


Barbara Japelj Pavešić - Paulina Koršñáková - 
Sabine Meinck 
Editors 


Dinaric Perspectives 
on TIMSS 2019 


Teaching and Learning Mathematics 
and Science in South-Eastern Europe 


E European B IEAA Springer 


Commission 


Editors 


Barbara Japelj Pavešié Paulina Koršñáková 
Educational Research Institute International Association for the Evaluation 
Ljubljana, Slovenia of Educational Achievement (IEA) 


Amsterdam, The Netherlands 
Sabine Meinck® 
International Association for the Evaluation 
of Educational Achievement (IEA) 
Hamburg, Germany 


ISSN 2366-1631 ISSN 2366-164X (electronic) 
IEA Research for Education 
ISBN 978-3-030-85801-8 ISBN 978-3-030-85802-5 (eBook) 


https://doi.org/10.1007/978-3-030-85802-5 


O International Association for the Evaluation of Educational Achievement (IEA) 2022. This book is an 
open access publication. 

Open Access This book is licensed under the terms of the Creative Commons Attribution-NonCommercial 
4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncom- 
mercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you 
give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons 
license and indicate if changes were made. 

The images or other third party material in this book are included in the book's Creative Commons license, 
unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative 
Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted 
use, you will need to obtain permission directly from the copyright holder. 

This work is subject to copyright. All commercial rights are reserved by the author(s), whether the whole 
or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, 
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or 
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar 
methodology now known or hereafter developed. Regarding these commercial rights a non-exclusive 
license has been granted to the publisher. 

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication 
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant 
protective laws and regulations and therefore free for general use. 

The publisher, the authors and the editors are safe to assume that the advice and information in this book 
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or 
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any 
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional 
claims in published maps and institutional affiliations. 


This Springer imprint is published by the registered company Springer Nature Switzerland AG 
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland 


Foreword I 


IEA’s mission is to enhance knowledge about education systems worldwide and 
to provide high-quality data that will support education reform and lead to better 
teaching and learning in schools. In pursuit of this aim, it conducts and reports on 
major studies of student achievement in literacy, mathematics, science, citizenship, 
and digital literacy. These studies, most notably TIMSS, PIRLS, ICCS, and ICILS, 
are well established and have set the benchmark for international comparative studies 
in education. 

The studies have generated vast datasets encompassing student achievement, 
disaggregated in a variety of ways, along with a wealth of contextual information that 
contains considerable explanatory power. The numerous reports that have emerged 
from them are a valuable contribution to the corpus of educational research. 

Valuable though these detailed reports are, IEA’s goal of supporting education 
reform needs something more: deep understanding of education systems and the 
many factors that bear on student learning advances through in-depth analysis of 
the global datasets. IEA has long championed such analysis and facilitates scholars 
and policymakers in conducting secondary analysis of our datasets. So, we provide 
software such as the International Database Analyzer to encourage the analysis of our 
datasets, support numerous publications including a peer-reviewed journal—Large- 
scale Assessment in Education—dedicated to the science of large-scale assessment 
and publishing articles that draw on large-scale assessment databases, and organize a 
biennial international research conference to nurture exchanges between researchers 
working with IEA data. 

The IEA Research for Education series represents a further effort by IEA to 
capitalize on our unique datasets and to provide powerful information for policy- 
makers and researchers. Each report focuses on a specific topic and is produced by 
a dedicated team of leading scholars on the theme in question. Teams are selected 
on the basis of an open call for tenders; there are two such calls a year. Tenders are 
subject to a thorough review process, as are the reports produced. (Full details are 
available on the IEA website.) 

The Dinaric region—the regional emphasis of the current volume—is named 
after the Dinaric Alps. Located in South-eastern Europe, this region stretches from 
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Slovenia through Croatia, Bosnia and Herzegovina, Serbia, Montenegro, Kosovo, 
Albania, and North Macedonia!. This thirteenth volume in the series features invited 
experts from across the region who offer a multidimensional and culturally sensi- 
tive perspective on the Dinaric TIMSS 2019 fourth grade results within the region. 
In addition, the volume is unique in that chapters that involve empirical analyses 
were completed in close collaboration with analysts from the IEA. The authors of 
this volume examine TIMSS participation in the region over time, the approaches 
of participating countries to implement TIMSS 2019 at the grade four level, and 
the wider educational contexts of the various systems, including demographic and 
cultural factors. Relevant within and beyond the Dinaric region, chapters in this 
volume include analyses of opportunity to learn in mathematics and science (Chapter 
“Opportunity to Learn Mathematics and Science”); student interests and beliefs 
(Chapter “Students’ Interests, Motivation, and Self-beliefs"); and characteristics of 
students, teachers, and their principals across the region (Chapters "Characteristics 
of High- and Low-performing Students” and “Scaffolding the Learning in Rural 
and Urban Schools: Similarities and Differences"). This book will be a valuable 
resource for researchers interested in this dynamic and culturally rich region of the 
world. Further, Dinaric educators will find the analyses and comparisons useful for 
understanding commonalities and differences across the region. 

Future volumes in the series include one dedicated to so-called process data— 
a by-product of a computerized testing platform—and one on the sustainability of 
global citizenship education. 


Series editors 
Seamus Hegarty 
Leslie Rutkowski 


' The volume includes analyses from all participating Dinaric education systems. Slovenia did not 
participate in the 2019 TIMSS cycle. 
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In February 2018, the European Commission presented the new Western Balkans 
Strategy, stressing the European future of the region as a geostrategic investment in a 
stable, strong, and united Europe based on common values. This was again confirmed 
at the Zagreb Summit in May 2020. 

Education, culture, youth, and sport, together with research and innovation, are 
recognized as essential drivers to boost the economic development, competitive- 
ness, and social cohesion of the region. This year we have learned that unexpected 
events and crises, as demonstrated by COVID-19, may challenge access to education. 
Ensuring access to quality education is imperative to give every child a better future; 
develop their full potential; and, ultimately, contribute to a peaceful, inclusive, and 
prosperous life for our societies. 

IEA’s Trends in International Mathematics and Science Study (TIMSS) 2019 
mark a milestone in terms of the participation of the Western Balkans region. For 
the first time, the entire region participated in the same TIMSS round of testing, 
promoting peer learning. This joint venture supports the ability of the countries in 
the region to learn from each other by identifying similarities and differences between 
the neighboring education systems. Further, the publication gathers together useful 
regional data that can support education authorities to develop appropriate measures 
to address the most pressing education needs. 

The enhanced focus on mathematics and science fits well with the European 
Commission’s renewed priorities on science, technology, engineering, and mathe- 
matics (STEM) and with our efforts to increase the number of women studying STEM 
subjects. These are also reflected in the Digital Education Action Plan 2021—2027 
(European Commission 2021) and in the Communication on Achieving the European 
Education Area by 2025 (European Commission 2020). 

Previous experience shows that TIMSS results enhance awareness and drive policy 
change. I trust this report will serve as a knowledge basis for all regional stakeholders, 
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among them teachers, school leaders, researchers, and decisionmakers, to support 
and enhance student achievement and engagement in mathematics and science. 
Enjoy reading the report. 


Themis Christophidou 
Director-General for Education, Youth, 
Sport, and Culture, European 
Commission 

Bruxelles, Belgium 
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Abstract Ensuring access to quality education is imperative to give every child a 
better future, develop their full potential and, ultimately, contribute to a peaceful, 
inclusive, and prosperous society. In 2019, for the first time, the entire Dinaric 
region participated in IEA’s Trends in International Mathematics and Science Study 
(TIMSS) at grade four, flagging a unique opportunity to gather robust comparative 
regional data about neighboring education systems. The international collaborative 
research presented in this book identifies both similarities and differences among 
these educational systems, sharing useful information about the relative strengths 
and weaknesses of each, designed to inspire peer learning and improve progress 
toward quality education for all. Secondary analyses of the TIMSS 2019 data provides 
some contextual understanding for education authorities across the Dinaric region, 
enabling them to review their own educational aspirations, connect their practical 
experience with empirical evidence, and together advance educational collaboration 
across the region. 


Keywords Albania : Bosnia and Herzegovina : Croatia : Kosovo : Montenegro : 
North Macedonia : Serbia - Grade four education : International large-scale 
assessments (ILSA) - Trends in International Mathematics and Science Study 
(TIMSS) 


P. Koršñáková (EX) 

International Association for the Evaluation of Educational Achievement (IEA), Amsterdam, The 
Netherlands 

e-mail: p.korsnakova @iea.nl 


S. Meinck 

International Association for the Evaluation of Educational Achievement (IEA), Hamburg, 
Germany 

e-mail: sabine.meinck ? iea-hamburg.de 


B. Japelj Pavešié 
Educational Research Institute, Ljubljana, Slovenia 
e-mail: barbara.japelj C pei.si 


O International Association for the Evaluation of Educational Achievement (IEA) 2022 1 
B. Japelj Pavešié et al. (eds.), Dinaric Perspectives on TIMSS 2019, IEA Research 
for Education 13, https://doi.org/10.1007/978-3-030-85802-5_1 


2 P. Koršñáková et al. 


1 Defining the Dinaric Region 


Definitions of Southeastern Europe are various, and may be disputed depending 
on the perspective, which can be political, economic, historical, cultural, and/or 
geographical. The same is true for the Balkans (a name derived from the Balkan 
Mountains), a geographic area in Southeastern Europe with various definitions and 
meanings, which include both the geopolitical and historical. Both terms commonly 
refer to a wider area that usually includes Albania, Bosnia and Herzegovina, Bulgaria, 
Croatia, Greece, Kosovo,! Montenegro, North Macedonia, and Serbia. Sometimes, 
Moldova, Romania, Slovenia, and the East Thrace (part of Turkey) are also included. 


Fig. 1 The “Western Balkan" region, comprising Albania, Bosnia and Herzegovina, Montenegro, 
North Macedonia, Serbia, and Kosovo. Croatia (indicated by hatching) joined the European Union 
in 2013 


! This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 
(United Nations, 1999) and the International Court of Justice (ICJ) Opinion on the Kosovo 
declaration of independence (ICJ, 2010). 
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Western Balkans is a political neologism that has been used to refer to Albania 
and the territory of former Yugoslavia, except Slovenia, since the early 1990s 
(Commission of the European Communities, 2008). The institutions of the Euro- 
pean Union (EU) have generally used the term “Western Balkans” (Fig. 1) to refer 
to the Balkan area that includes non-members of the EU, and developed a policy to 
support the gradual integration of these Western Balkan economies into the Union. 
On 1 July 2013, Croatia became the first of this group to join the EU, and Montenegro, 
Serbia, North Macedonia, and Albania are official candidates for membership. Acces- 
sion negotiations and chapters have been opened with Montenegro and Serbia, and 
Bosnia and Herzegovina and Kosovo are potential candidates for future membership 
(European Parliament, 2019). 

The region of the Western Balkans, as used in the European political context, 
roughly corresponds to the territory of Dinaric Alps (or Dinarides), also known as the 
Alpet Dinaride or Alpet Dinarike in Albanian and Dinaridi//Iugapu in Bosnian, 
Croatian, and Serbian. They are named after Mount Dinara (1831 m), which lies in 
the center of the mountain range located at the border of the Dalmatian part of Croatia 
with Bosnia and Herzegovina, and stretches through Slovenia, Croatia, Bosnia and 
Herzegovina, Serbia, Montenegro, and Kosovo to Albania in the southeast. The 
Dinaric Alps extend south to the Sharr Mountains, which connect Kosovo and the 
northwest of North Macedonia to northeastern Albania (Fig. 2). 

Comprising an area of approximately 100,000 km2, the Dinaric Alps stretch along 
more than 6000 km of coastline, including the entire area facing the Adriatic Sea, and 
naturally connect Croatia, Bosnia and Herzegovina, Montenegro, Serbia, Kosovo, 
Albania, and North Macedonia. This region will be the focus of this publication. The 
geographical units share many common cultural elements, and the characteristics 
of natural environment are similar, but they differ in size and population. Kosovo 
and Montenegro are geographically the smallest, being 10,887 km? and 13,810 km?, 
respectively, while Serbia is geographically the largest at 88,360 km?. 

Population sizes range from c. 600,000 inhabitants in Montenegro to a population 
of c. seven million in Serbia, as of 2018. Kosovo and Albania have the highest 
population densities, with c. 168 people per km? and 105 people per km2, respectively. 
In general, at least half of the populations in the region live in urban areas, ranging 
from up to 67% in Montenegro to only 48% in Bosnia and Herzegovina.” In most of 
the economies of the region, the percentage of the population living in urban areas 
increased slightly during 2018, except in Bosnia and Herzegovina, Croatia, and 
Serbia. The life expectancy at birth in the region lies between 72.2 years (Kosovo) 
and 78.07 years (Croatia). The gross national income (GNI) per person (in terms 
of purchasing power parity) for 2018 varied between US$ 11,540, in Kosovo, and 
US$ 27,180, in Croatia (World Bank, 2020). 

According to the United Nations Development Programme's (UNDP, 2019) 
human development index (HDI; a composite of indicators for a long and healthy 
life, knowledge, and a decent standard of living), the seven participants involved in 


? There is no information available about the urban population in Kosovo. 
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app. borders of the Dinaric Alps @ Southwestern belt (Maritime Dinaric Alps) 
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Fig. 3 The Dinaric Alps 


this study? were ranked as lying between 46th (Croatia) and 82nd (North Macedonia) 
among 189 countries in 2018, and all had shown continuous improvement in their 
scores since they were included in this UNDP index. 


2 Trends in International Mathematics and Science Study 
at Grade Four 


EA’s Trends in International Mathematics and Science Study (TIMSS) measures 
student achievement in the subjects of mathematics and science every four years by 
administering tests to a sample of students at the specified grade (for this research, we 
focused on grade four). By using advanced sampling methodology, TIMSS ensures 
a representative sample of the student population in each participating education 
system. Background information is collected from sampled students, their school 
principals, teachers, and parents, and includes factors that affect learning, including 


3 Kosovo is not included in the HDI ranking. 
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school resources, student attitudes, instructional practices, and support at home. The 
TIMSS results and further analyses of the background information may therefore 
provide discoveries that can be used to inform future education policy and practices 
around the world. The TIMSS design enables the measurement of trends in educa- 
tional achievement, across evolving contexts and reformed educational provision 
over years and across countries. Advanced statistical modeling of the measurement 
of achievement ensures that results can be compared with previous cycles, although 
the set of participating countries and test materials administered changes from one 
cycle to the next (Martin et al., 2020). 

TIMSS is grade-based and curriculum-rooted, and, in this region, the research 
interest on grade four coincides with a cohort of approximately 10-year-old students 
at the primary school level. TIMSS considers the context, examining processes as 
well as outcomes of education, in order to understand the linkages between the 
intended curriculum (what policy requires), the implemented curriculum (what is 
taught in schools), and the achieved curriculum (what students learn). The concept 
of “opportunity to learn” is the underlying focus of the study model, expressed by the 
framework that serves as the basis for the instrument development and data collection 
(Mullis & Martin, 2017). 

National research coordinators (NRCs) ensure that study instruments and proce- 
dures are appropriate for their students and suit the educational context of their 
system. Assessment questions are pre-tested (this is referred to as “pilot” and “field” 
testing), and any issues identified during these early trials are addressed before the 
main assessment is administered. IEA makes every effort to safeguard the quality and 
comparability of data through careful planning and documentation, supporting coop- 
eration among participating education systems, standardization of procedures, and 
rigorous quality control (Martin et al., 2020). The resulting data are organized and 
stored in an international database, ensuring full comparability across countries and 
with data from previous years. Datasets, complemented by detailed technical docu- 
mentation and user guides (Fishbein et al., 2021) are available as free open-access 
resources for research on the websites of IEA (2021a), and the TIMSS & PIRLS 
International Study Center at Boston College (TIMSS & PIRLS International Study 
Center, 2021). 

Two decades of TIMSS results (1995-2015) reveal important trends. For example, 
more countries have registered increases rather than decreases in average student 
achievement scores for grade four mathematics and science. Students have also 
demonstrated increasing levels of knowledge, and gender gaps in student achieve- 
ment are decreasing. These overall improvements in educational achievement are 
accompanied by additional gains, such as improved school environments (e.g., 
safer schools), better educated teachers, more support for teachers” professional 
development, and better curriculum coverage (Mullis et al., 2016). 

The TIMSS open access datasets are recognized by the United Nations Educa- 
tional, Scientific and Cultural Organization (UNESCO) as a solid evidence base for 
researchers, educators, and policymakers interested in monitoring progress toward 
the sustainable development goals (SDGs) (UNESCO Institute for Statistics, 2018). 
The lowest of the four TIMSS international benchmarks, which serve as specific 
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points on the scale of measured achievement, represents a level of basic knowledge 
and competencies equivalent to the “SDG minimum proficiency level." On average, 
across all TIMSS 2019 countries, 92% of students met this level of basic knowledge 
in TIMSS 2019 grade four mathematics, showing that they could add and subtract 
whole numbers, had some understanding of multiplication by one-digit numbers, 
could solve simple word problems, and had some knowledge of simple fractions, 
geometric shapes, and measurements; the percentage of grade four students achieving 
this level of competency in mathematics varied across the Dinaric region from 73 to 
95% (Mullis et al., 2020). Meanwhile, 92% of grade four students across all TIMSS 
2019 countries met the TIMSS 2019 minimum proficiency levelin science by demon- 
strating that they had some basic knowledge of scientific concepts and foundational 
scientific facts. The percentage of grade four students achieving this level of scientific 
understanding varied from 59 to 98% across the participating Dinaric region systems 
(Mullis et al., 2020). 


3 Engagement with TIMSS 


The Dinaric region first became involved in IEA international assessments over 
60 years ago; the former Yugoslavia was one of the countries that participated 
in IEA's Pilot Twelve-Country Study project in 1959-1961 (IEA, 2021b). This 
project assessed five subject areas: mathematics, reading comprehension, geography, 
science, and non-verbal ability (Foshay et al., 1962). The six successor states of 
Yugoslavia, namely Bosnia and Herzegovina, Croatia, Montenegro, North Mace- 
donia, Serbia, and Slovenia, have also participated in various TIMSS cycles. For 
example, Bosnia and Herzegovina participated at grade eight in TIMSS 2007, Croatia 
at grade four in TIMSS 2011 and 2015, North Macedonia at grade eight in TIMSS 
1999, 2003, and 2011, Serbia at grade eight in TIMSS 2003 and 2007 and at grade 
four in 2011 and 2015, and Slovenia at grades four and eight in all cycles from 1995 
to 2015. However, TIMSS 2019 marks a unique milestone for participation in the 
region, with seven participants administering the study at grade four. 

The TIMSS results have been used in different ways by education systems across 
the region, and so have had varying impact. For example, TIMSS was adminis- 
tered at grade eight in Bosnia and Herzegovina in 2007, and this was followed by 
a national secondary analysis of the data two years later. Results from this analysis 
were made publicly accessible to local stakeholders. While education authorities did 
not use these results to shape educational policies, a few enthusiastic teachers and 
other professionals did make use of the outcomes (Centre for Policy and Gover- 
nance, 2013; Popić & Džumhur, 2020; Suzić & Ibrakovié, 2009). However, TIMSS 
results have also contributed toward major policy changes in the region. In Croatia, an 
expert group, supported by various stakeholders in the field, used results from interna- 
tional large-scale studies (together with other data) as foundation for the most recent 
curriculum reform, which was launched with the school year 2019/2020. TIMSS has 
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thus had direct impact on national policies and educational reforms, with perma- 
nent influence on teaching practices for primary education in Croatia. Specifically, 
secondary analysis of TIMSS 2015 results resulted in changes in the curriculum for 
the subjects mathematics, physics, chemistry, biology, and nature and science. For 
example, for physics, the TIMSS 2015 analysis revealed that Croatian students had 
only limited familiarity with some of the test content, as it was either entirely absent 
from the national curriculum or was taught at a higher grade. As a direct consequence 
of these findings, the revised 2019/2020 physics curriculum introduced new content 
areas, moved content to earlier grades, or upgraded content areas to higher levels, 
such as understanding or connecting concepts. The Croatian Education and Teacher 
Training Agency also used the results to develop a series of teacher training courses 
about TIMSS. TIMSS results were thus used to prompt discussion about the learning 
and teaching challenges evolving from the paradigm change from traditional ways of 
reproducing theoretical knowledge towards new approaches for developing student 
competencies (Elezović & Muraja, 2020). 

In North Macedonia, participation in TIMSS contributed to a higher awareness 
of the generally low level of national student achievement and the need for external 
measurement. The results were used to develop new curricula for mathematics, chem- 
istry, physics, and biology, and prompted the introduction of a new science subject 
named natural sciences, which is now taught nationally from grade one to grade six 
(Lameva, 2020). 

TIMSS has also influenced educational policies in Serbia since 2003. Serbian 
educational authorities recognize TIMSS study results as an indicator of the effec- 
tiveness of the whole education system and use them as basis for decision-making 
to improve the quality of education. TIMSS results also contributed to the develop- 
ment of educational standards for mathematics and science in primary education; this 
can be considered one of the most important outcomes of the study. Furthermore, 
the experience of participating in TIMSS was used as a basis for preparation of 
procedures for the end of school examinations and for national testing. The Serbian 
education authorities have also used data collected by TIMSS on school infrastruc- 
ture to make decisions about future school investments, as well as using selected data 
and materials as supporting materials in teacher education programs (Đerić et al., 
2020; Kadijevié et al., 2004; Kadijevich, 2019). 


3.1 An Aside: Slovenia's Participation in TIMSS 


While Slovenia did not participate in TIMSS 2019 for financial reasons, it had previ- 
ously participated from 1995 until 2015. In Slovenia, TIMSS was regarded as a 
reliable standard tool for measurement of mathematics and science education. The 
unbiased perspective of the reports from international comparisons was recognized 
as constructive, avoiding some of the direct criticisms directed at national projects 
focused on nationally known problems. TIMSS results were used to initiate changes 
in many areas of the educational system. For example, results were used to introduce 
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new content (i.e., data displays) into the curricula and alter the order of teaching 
science content in early grades, and sharing international comparisons of time spent 
on learning and homework helped to change public opinion on what was an appro- 
priate student workload. TIMSS data also became an important information source 
for national proJects, providing information on, for example, regional differences, 
or gender and age gaps in achievement. Teachers were encouraged to use publicly 
available items from TIMSS in their teaching practice, and use these to design simi- 
larly challenging items or connect different content. TIMSS cognitive areas totally 
changed the understanding of mathematics and science cognitive levels. Teachers 
learned that attitudes have an important role in teaching and became attentive to 
background factors linked to achievement, teaching, and learning (Japelj Pavešič, 
2013). Mathematics and science achievement increased over time in Slovenia, and 
national conferences providing extended feedback to teachers about student success 
resulted in improvements in teaching practice. 


4 Aspirations and Expectations for This Book 


The examples in Sect. 3 demonstrate how TIMSS results have been used to inform 
educational authorities and stakeholders in the field of education, support decision- 
making, and guide educational reforms in the region. Education systems can benefit 
from the high-quality data retrieved from standardized large-scale assessments. Such 
data enable secondary analyses that may shed light on specific education-system- 
level questions or issues, which together with educational stakeholder engage- 
ment and reflection, results in better understanding and evidence-driven action. The 
secondary analyses based on the TIMSS 2019 cycle data from neighboring educa- 
tion systems provide educational authorities across the Dinaric region with additional 
tools to review their own education systems' strengths and weaknesses. With this 
much deeper contextual understanding, they can connect practical experiences in the 
region with empirical evidence from TIMSS 2019. This book provides an initial inter- 
pretation of the regional educational landscape in 2019, and the analyses we present 
are designed to prompt researchers to investigate other aspects of their education 
systems. 


5 Notes About the Statistical Analyses Methods Used 
in This Book 


To compare findings across the Dinaric region, we analyzed data using basic and 
advanced methods to estimate percentages, means, correlations, and develop regres- 
sion models. We conducted all statistical computations using established standard 
procedures for data from large-scale assessments. For all our calculations, we used the 
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IEA’s IDB (International Database) Analyzer (IEA, 2021a), a statistical tool specif- 
ically developed for the correct analysis of large-scale assessment data that works 
in conjunction with the well-known SPSS statistical package (IBM, 2021). This 
tool accounts for the complex unit and item sampling design by applying sampling 
weights to the analyses, and uses plausible values when working with achievement 
variables. We used the jackknife repeated replication method (as described in Martin 
et al., 2020) to determine standard errors and statistics related to significance tests 
of group differences or other statistical parameters (e.g., correlation and regression 
coefficients). The IDB Analyzer tool was used to calculate Pearson correlation coef- 
ficients for selected variables of interest (Freedman et al., 2007; see also IEA, 2021c 
for instructional videos on the use of the IDB Analyzer, including one covering 
Pearson correlation coefficients). Throughout, we used t-test statistics to determine 
statistical group differences, assuming two-tailed tests with a significance level of 
a = 0.05. 

In TIMSS, items assessing a common underlying construct are combined to form 
a scale. The individual scales used in the chapters of this book are available in the 
TIMSS international database (TIMSS & PIRLS International Study Center, 2021) 
and their construction is described in detail in the TIMSS technical report (Yin & 
Fishbein, 2020). These TIMSS scales are constructed using item response theory 
scaling methods, with a scale center point of 10 (to represent the mean score of 
the combined distribution of all TIMSS 2019 grade four participants). In each case, 
the units of the scale are chosen so that the standard deviation of the distribution is 
equivalent to two scale score points. All cases with valid responses to at least two 
items on a scale were included in the calibration and scoring processes. Each scale 
was divided into three regions (representing high, middle, and low score values) 
designed to provide a content-referenced interpretation for the scale values. The 
boundaries between scale score regions differ across attitude scales; the cut points 
were based on judgments made by the TIMSS & PIRLS International Study Center 
staff and are presented together with the scales’ means for each TIMSS participating 
education system in the TIMSS 2019 international reports (Mullis et al., 2020; Yin 
& Fishbein, 2020). 

Although we provide references in each chapter, we encourage readers interested 
in data availability and quality issues, or further general information about TIMSS 
2019, to explore the following publications and resources: 


(1) The TIMSS 2019 Assessment Frameworks describe the general foundations of 
mathematics and science assessment, as well as the additional factors asso- 
ciated with student learning in mathematics and science that are investigated 
using the TIMSS questionnaires completed by students, their parents, teachers, 
and school principals. It also provides an overview of the assessment design, 
including general parameters for item development (Mullis & Martin, 2017). 

(2) The TIMSS 2019 Encyclopedia is a comprehensive compendium of how math- 
ematics and science are taught in the education systems participating in the 
study. Each TIMSS 2019 participant prepared a chapter summarizing the key 


10 P. Koršñáková et al. 


aspects of mathematics and science education within their education system 
and answered the TIMSS 2019 curriculum questionnaire (Kelly et al., 2020). 

(3) The TIMSS 2019 International Results in Mathematics and Science summa- 
rizes a wide array of results, including achievements and trends (Mullis et al., 
2020). 

(4) Methods and Procedures. TIMSS 2019 Technical Report provides additional 
details related to the development of the TIMSS assessments and question- 
naires, the documentation of the numerous quality assurance steps and proce- 
dures implemented by all those involved in the TIMSS 2019 assessments, and 
also describes the methods used for sampling, translation verification, data 
collection, database construction, and the construction of the achievement and 
context questionnaire scales (Martin et al., 2020). 

(5) The TIMSS 2019 User Guide for the International Database supports and 
facilitates the use of the data collected in TIMSS 2019. As mentioned in Sect. 2, 
a public-use version of the database is available for download from IEA and the 
TIMSS & PIRLS International Study Centre at the Boston College (Fishbein 
et al., 2021). 


6 Overview of the Chapter Contents 


Dinaric Perspectives on TIMSS 2019 uses secondary analyses of the TIMSS data 
to develop a multidimensional, context-rich perspective on TIMSS results at grade 
four for seven participants from the Dinaric region. Data from Albania, Bosnia and 
Herzegovina, Croatia, Kosovo, Montenegro, North Macedonia, and Serbia provide 
a basis for the comparison of the different contexts for learning and methods for 
teaching science and mathematics to grade four students. 

Chapter “Context and Implementation of TIMSS 2019 at Grade Four in the Dinaric 
Region” examines the implementation of TIMSS 2019 across the region, exploring 
the different education systems and study-specific context information, highlighting 
both regional similarities and differences. This includes the structure of the formal 
schooling systems, starting with early childhood education and care, to the end of the 
years of compulsory education, outlining the language of instruction, the mathematics 
and science curricula, and quality assurance components across the region. The 
chapter also addresses certain TIMSS administration procedures, such as sampling 
or test administration, ensuring the context of our findings is clearly understood. 

Chapter “Opportunity to Learn Mathematics and Science” examines opportu- 
nity to learn mathematics and science, or the “observable structure” of teaching for 
learning outcomes, which includes the intended, implemented, and attained curricula. 
Specifically, this chapter investigates the relationship between the content taught and 
student achievement across education systems. 

Students’ interests, motivation, and self-beliefs and their impact on student 
achievement are the focus of chapter “Students’ Interests, Motivation, and Self-be- 
liefs". Student achievement has been linked to student attitudes toward learning about 
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mathematics and science, student motivation, and confidence, as well as parental 
attitudes toward mathematics and science. 

Chapter “Early Literacy and Numeracy Competencies: Predictors of Mathematics 
Achievement in the Dinaric Region” looks at early literacy and numeracy compe- 
tencies in the Dinaric region. Factors such as socioeconomic status, the number of 
years spent in early childhood education facilities, and home resources have all been 
associated with early literacy and numeracy competencies, which are, in turn, related 
to student performance in schools. The chapter analyzes regional differences in these 
competencies and how they are related to student achievement. 

Chapter “The Role of Learning Resources, School Environment, and Climate 
in Transforming Schools from Buildings to Learning Communities” examines the 
role of learning resources and school environment in transforming schools from 
buildings to learning communities. Identifying the characteristics of school resources 
and environment that create successful school environments may ameliorate the lack 
of resources on a school or individual level. The analyses also cover the relationship 
between school emphasis on academic success and student achievement and the 
relationship between students’ sense of belonging and achievement results. 

Chapter “Teachers, Teaching and Student Achievement” explores the relationship 
between the quality of teachers (measured in terms of education and professional 
development), instructional practice in participating classes, and grade four student 
outcomes on the TIMSS test. In the Dinaric region, grade four students have teachers 
with similar educational backgrounds (in terms of experience, level of education, and 
level of professional development). Robust regional analyses supply an evidence base 
for future investigation into the effectiveness of the strategies for improvement. 

Chapter “Characteristics of Principals and Schools in the Dinaric Region” exam- 
ines school effects on the academic achievement of students. The research looks at 
whether the level of education, years of experience of the principal, the location of the 
school, and/or school composition have significant effects on student achievement, 
as well as perceptions of school emphasis on academic success. 

After defining high- and low-performing students, according to the proficiency 
levels set by the TIMSS international benchmarks in mathematics and science 
for both groups, chapter “Characteristics of High- and Low-performing Students” 
describes and compares selected characteristics of these groups of students across 
the region. 

Finally, chapter “Scaffolding the Learning in Rural and Urban Schools: Similari- 
ties and Differences” identifies differences and similarities between rural and urban 
schools, particularly from the perspective of different types of support for student 
learning. A better understanding of the urban-rural achievement gap in science and 
mathematics, taking into account family and school factors, may improve support 
for learning at school. 
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Context and Implementation of TIMSS A) 
2019 at Grade Four in the Dinaric Region 


Paulína Koršňáková and Sandra Dohr 


Abstract Governments and policymakers around the world view large-scale assess- 
ments, such as IEA' Trends in International Mathematics and Science Study 
(TIMSS), as increasingly important in supporting continuous improvement in the 
quality of education and education systems. However, a good knowledge of the 
contexts and environments that exist when these studies are implemented is essential 
to fully understand study results, draw conclusions, and make policy recommenda- 
tions. Collecting background information about the components of the participating 
education systems thus supports better appreciation and interpretation of the TIMSS 
2019 assessment results for the Dinaric region. Relevant topics include descriptions of 
the early childhood education and care (ECEC) sector, compulsory education provi- 
sion, languages of instruction across the region, and brief summaries of the existing 
mathematics and science curricula. The role of assessments for quality assurance 
in the region, and an introduction to the TIMSS 2019 survey design and its local 
implementation, provide additional context. Together, this supporting documenta- 
tion gives crucial insight into how education functions throughout the region and 
serves as solid basis for interpreting the analyses in this book. 


Keywords Assessment - Compulsory education - Digitalization * Dinaric region : 
Early childhood education - Mathematics instruction * Primary school + Quality 
assurance + Science instruction * Trends in International Mathematics and Science 
Study (TIMSS) - Western Balkans 


1 Introduction 


Participation in international large-scale assessments enables education systems to 
compare and learn from each other; the results are used as a means to improve 
educational quality. However, to interpret the outcomes, it is important to consider 
the various, and likely unique combination of antecedent contextual and explanatory 
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factors that also play a role in the assessment results. Education systems serve social 
and cultural goals (Komatsu & Rappleye, 2017) and reflect the overall economic situ- 
ation that interconnects governmental decisions related to specific educational goals 
with politics and resources (see, e.g., Hanushek & Woessmann, 2019). Looking into 
educational policies and practices can provide essential context for understanding 
student performance results from large-scale assessments, such as IEA’s Trends in 
International Mathematics and Science Study (TIMSS), and can eventually contribute 
to improving results over time. Establishing the context for teaching and learning is 
therefore the basis for analyzing data and making comparisons between education 
systems. 

Seven participants from the Dinaric region took part in TIMSS 2019, namely 
Albania, Bosnia and Herzegovina, Croatia, Kosovo, ! Montenegro, North Macedonia, 
and Serbia. We provide a comparative overview of the structures of the participating 
education systems and a brief introduction to some of the topics that are covered 
in depth in this book. We first explore the paths of schooling that children in the 
region typically go through during their school career, including opportunities for 
early childhood learning. We then provide a brief synopsis of the mathematics and 
science curricula at the system level. A particular focus of this chapter is how quality 
assurance functions in each of the respective education systems and what role student 
assessments play in that context. We conclude by examining the most important 
features of TIMSS 2019 and the implementation of the study at grade four across 
the region. 


2 Schooling Paths in the Region 


2.1 Early Childhood Education and Care 


Generally, the term early childhood education and care (ECEC) refers to the 
“provision for children before the start of compulsory primary education” 
(European Commission/EACEA [Education, Audiovisual and Culture Executive 
Agency ]/Eurydice, 2019c, p. 12) and includes services that ensure the child's safety 
and care as well as services that support the child's educational development. The 
ECEC sector can be further categorized, and the transition from a childcare-type 
setting (e.g., nurseries) to an education-type setting (e.g., kindergartens), or the age 
of transition (under and over three years), are commonly used to divide the sector 
(European Commission/EACEA/Eurydice, 2019c). 

ECEC provision is generally available in form of nurseries, kindergartens, and pre- 
school preparatory programs in the Dinaric region, and attendance at this level is not 
mandatory, except in parts of Bosnia and Herzegovina, Croatia, and Serbia. Bosnia 


! This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 
(United Nations, 1999) and the International Court of Justice (ICJ) Opinion on the Kosovo 
declaration of independence (ICJ, 2010). 
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and Herzegovina has structured their provision into three different levels, which 
include nurseries for children between six months and three years, kindergartens for 
children from three to six years, and pre-school preparatory programs, which start 
a year before the first grade of primary school. The last level is compulsory for the 
majority of the population (Popić & Džumhur, 2020). Serbia provides pre-school 
education in form of nurseries for children between six months and three years, 
and kindergartens for children aged three to seven. Since the school year 2006, one 
year of preparatory pre-school education before the beginning of primary school has 
become mandatory (Deri¢ et al., 2020). In Croatia, attending the last year of ECEC in 
part-time format was made compulsory in 2014. This implies that children in Croatia, 
Serbia, and parts of Bosnia and Herzegovina are legally entitled to have a guaranteed 
place for the last year of ECEC (European Commission/EACEA/Eurydice, 2019c). 

Participation rates for the ECEC sector are available for the region from different 
sources (e.g., both Eurostat, the European Commission’s statistics hub, and UNICEF, 
the United Nations Children’s Fund, publish such statistics). However, depending on 
the methodology of national statistical institutes, the definition of an ECEC insti- 
tution, or the classification of age groups, numbers can vary considerably between 
education systems and thus, comparisons may be difficult. For instance, participa- 
tion rates could include only children in early childhood educational development 
programs (coded as International Standard Classification of Education [ISCED] 010; 
see UNESCO [United Nations Educational, Scientific and Cultural Organization] 
Institute of Statistics, 2012 for an explanation of the ISCED classifications) and 
exclude children in childcare provisions that fall outside of the ISCED classification 
or vice versa (European Commission/EACEA/Eurydice, 2019c). Another factor that 
needs to be taken into account is the structure of the sector and whether the offer 
is mainly public or private. This can have an effect on official participation rates 
(UNICEF, 2017). 

According to Eurostat data, participation rates for children under the age of three 
in ECEC provisions are only available for some education systems in the region; 
for 2017, these included Croatia (15.9%), North Macedonia (10.3%), and Serbia 
(14.5%). In 2017, participation rates of children between the age of four and the 
starting age of compulsory primary education were 82.8% in Croatia, 70.4% in 
Montenegro, 68.6% in Serbia, and 39.5% in North Macedonia (European Commis- 
sion/EACEA/Eurydice, 2019c). Even though enrollment rates in the ECEC sector 
are rather low throughout the region, it is notable that considerable efforts have been 
made in recent years to increase the participation of children in ECEC provision one 
year prior to the beginning of primary education, notably in Kosovo (Aliu, 2019; 
UNICEF, 2017) and Bosnia and Herzegovina (UNICEF, 2020). 


2.22 Compulsory Education 


Children in the Dinaric region generally enter primary school in the calendar year 
of their sixth or seventh birthday, depending on the various regionally established 
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Table 1 Students’ starting and leaving age for compulsory education/training and duration 


Education system Full-time compulsory education/training 
Starting age (years) Leaving age (years) | Duration (years) 
Albania 6 15 9 
Bosnia & Herzegovina 6 15 9 
Croatia? 7 15 8 
Kosovo^ 6 14 9 
Montenegro 6 15 9 
North Macedonia 5 yrs and 7 months 17, or 19 yrs and 11-13 
6 months 

Serbia 5 yrs and 6 months 14 yrs and 6 months | 9 
Source European Commission/EACEA/Eurydice (20192) 


Notes 

aThe year of compulsory pre-primary education in Croatia is only part-time education, and was 
therefore not considered in this table 

bData for Kosovo were obtained from the Division for Quality Assurance, Standards, Assessment 
and Licensing/Ministry of Education, Science, and Technology (DQASAL/MEST, 2020) 


cut-off dates. In Croatia, only the children who reach their sixth birthday before 1 
April can start school in September of that same calendar year, while, in Serbia, the 
equivalent cut-off date is 1 March. In addition to official policies, some education 
systems allow a degree of parental discretion and choice, or include a medical and 
psychological examination procedure in the enrollment process (Kelly et al., 2020). 
Considering compulsory education as “a period of full-time education/training that 
is compulsory for all students" (European Commission/EACEA/Eurydice, 2019a, 
p. 5), the duration of compulsory education in the region ranges from eight years in 
Croatia to 11—13 years in North Macedonia (see Table 1). The duration of compulsory 
education in North Macedonia depends on the chosen secondary school track, which 
can be general or vocational, and takes two to four years to complete (Lameva, 2020). 

Primary and lower secondary education is organized as a single-structure model 
across the region, with no transition between primary and lower secondary education 
(see Fig. 1) and includes general and compulsory education for all students. In addi- 
tion to some differences with regards to the entry age mentioned above, the duration 
of the primary level differs. Grade four may have a different position within the 
single structure. While grade four is the final grade of the first educational cycle in 
Croatia and Serbia, it is grade five in Albania, Bosnia and Herzegovina, and Kosovo. 
In Montenegro, and North Macedonia the primary and lower secondary education is 
organized in three-year cycles, grade four being the lowest grade of the second cycle 
(European Commission/EACEA/Eurydice, 2018). 


Context and Implementation of TIMSS 2019 at Grade Four ... 19 


m Single structure 
(ISCED 1 + ISCED 2) 


Common core curriculum 
(ISCED 2) 


© Differentiated 
branches/streams 
(ISCED 2) 


. i 
ess ex 


Fig.1 Main models of primary and lower secondary education (ISCED 1—2) in Europe 2018/2019 
Source Based on information provided by European Commission/EACEA/Eurydice (2018) 


2.3 Language of Instruction 


Across the region, several different languages are used for teaching in schools. The 
TIMSS 2019 assessment was administered in the main languages of instruction in 
the region (see Sect. 5, Table 2). 

In Croatia, Croatian is the only language of instruction in the great majority 
of the schools, however, in certain regions, minority languages are recognized as a 
second official language. Minorities are guaranteed education in their native language 
using three different approaches; namely, schools where all classes are taught in the 
minority language, schools where both Croatian and the minority language are taught, 
and schools where at least some additional language classes in the minority language 
are available. The recognized minority languages in Croatia are Albanian, Czech, 
Hungarian, Italian, German, Macedonian, Polish, Serbian, Russian, Rusyn, Slovak, 
Slovene, and Ukrainian (Elezović & Muraja, 2020). 
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Serbia uses the Serbian language as their main language of instruction, but ethnic 
minorities may receive instruction in their mother tongue. So far, instruction in 
primary schools has been organized in Albanian, Bosnian, Bulgarian, Croatian, 
Hungarian, Romanian, Rusyn, and Slovak (Derié et al., 2020). 

In Bosnia and Herzegovina, education is delivered in the official languages 
Bosnian, Croatian, and Serbian, depending on the curricula. In addition, some private 
schools offer instruction in English, French, Turkish, and German (Popié & DZumhur, 
2020). 

In North Macedonia, the official language of instruction is Macedonian, however, 
national minority groups are entitled to receive compulsory education in their 
mother tongue. Instruction in primary schools is available in Macedonian, Albanian, 
Bosnian, Serbian, and Turkish, whereas secondary schools only provide instruction 
in Macedonian, Albanian, and Turkish (Lameva, 2020). 

The official language of instruction in Montenegro is Montenegrin. However, 
teaching is also performed in other languages in official use, which include Serbian, 
Bosnian, Croatian, and Albanian (Radović, 2020). 

Kosovo has two official languages that are used for instruction in schools, namely 
Albanian and Serbian. In addition, national minorities have a right to education in 
their native language, such as Bosnian and Turkish (DQASAL/MEST, 2020). 

The majority of students in Albania are taught in Albanian. Recognized national 
minorities may receive schooling in their native language, which is currently available 
for the Greek and North Macedonian ethnic minorities (Council of Europe, 2017). 


3 Mathematics and Science Curricula in the Region 


In general terms, a curriculum can be defined as “the major organizing concept in 
considering how educational opportunities are provided to students and the factors 
that influence how students use these opportunities" (Mullis & Martin, 2017, p. 4). 
More precisely, the curriculum can be presented at three levels: (1) the intended; 
(2) implemented; and (3) attained curriculum. These levels, arranged in a top-down 
order, represent: (1) educational policies and national and social contexts; (2) actual 
teaching in the classroom, which includes home and school context; and (3) learning 
outcomes. To make improvements in student attainment, authorities in the educa- 
tion systems need to ensure that any intended changes are being implemented in 
schools and classrooms. Usually, it is a time-consuming effort to change routine and 
habit, which includes learning infrastructure and teaching practices. Quality assur- 
ance activities in the form of external evaluation, particularly if all stakeholders are 
involved, can aid with the implementation of the intended plans and improvements. 
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3.1 Teaching and Instruction 


A national curriculum that covers mathematics and science instruction at grade four 
is commonly used across the region, except in Bosnia and Herzegovina, where 
mathematics and science curricula vary among its constituents (however, in 2018, 
a new common core curriculum based on learning outcomes was established as a 
common foundation from which to create the individual curricula; Bosnia and Herze- 
govina Council of Ministers, 2018). For the remainder of the region, the mathematics 
curriculum that was in place during the TIMSS 2019 administration was introduced 
in 2000 in Kosovo, in 2006 in Croatia, in 2007 in Serbia, in 2014 in Albania, in 
2015 in North Macedonia, and in 2017 in Montenegro. Across the region, the science 
curriculum was generally introduced at the same time as the mathematics curriculum, 
except in Serbia, where the science curriculum was introduced in 2006, a year earlier 
than the mathematics curriculum (Kelly et al., 2020). During 2014-2016, North 
Macedonia adopted an adapted version of the Cambridge International Assessment 
curriculum (see Cambridge Assessment, 2021) for mathematics and science, which 
provides a comprehensive set of progressive learning objectives and a structure for 
teaching across grades one to nine (Lameva, 2020). In Croatia, Kosovo, and Serbia, 
the mathematics and science curricula have undergone continuous reform since the 
establishment of the curricula that were in place during the TIMSS 2019 administra- 
tion. During the 2018/2019 academic year, a new curricular reform was introduced 
on an experimental basis to a limited number of schools in Croatia, and this imple- 
mentation was subsequently extended to all schools in the following school year. The 
gradual introduction of the new curriculum, starting at grade one, means that reform 
of the grade four curriculum is scheduled for completion by school year 2022/2023 
(Elezovié & Muraja, 2020). In Kosovo, a new competency-based curriculum was 
initiated in the school year 2017/2018 at the school entry grades and its further 
implementation is still ongoing (DQASAL/MEST, 2020). 

For all TIMSS participants in the region, both mathematics and science are taught 
by the classroom teacher (see chapter “Teachers, Teaching and Student Achieve- 
ment” for more information on teachers’ instructional practice and measures of 
teacher quality). Across the Dinaric region, 17—22% of the total instruction time 
in grade four is devoted to teaching and learning mathematics, and nine to 17% of 
the time is allocated to teaching and learning science (see chapter "Opportunity to 
Learn Mathematics and Science" for more information on opportunities to learn). To 
evaluate the implementation of their mathematics and science curricula, all educa- 
tion systems make use of school inspector visits and school self-evaluation. Albania, 
Croatia, Kosovo, and Montenegro use national or regional examinations to eval- 
uate the (achieved) mathematics curriculum, and Kosovo also undertakes a research 
program for this purpose. Albania, Bosnia and Herzegovina, Croatia, Kosovo, and 
Montenegro use national or regional examinations to evaluate the science curriculum 
(Kelly et al., 2020). 
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3.2 Digital Competence and Use of Information 
and Communications Technology 


From the perspective of national mathematics curricula in the region, Albania, 
Kosovo, Montenegro, and North Macedonia include statements or policies about 
students’ use of digital devices in grade four mathematics instruction. For instance, 
in Montenegro, information and communications technology (ICT) use is frequently 
recommended for undertaking calculations and researching new ideas. Calculators 
are used in grade four mathematics instruction to check the correctness of calcula- 
tions. Acquiring ICT skills, such as finding, processing, and saving information, is 
also a general goal of the state’s education program, although none of the region’s 
education systems include specific statements about students’ use of digital devices 
during mathematics tests or exams in their curricula. However, the recently imple- 
mented curriculum in Croatia (which was initiated for grade one in the school year 
2018/2019) emphasizes using digital technologies in teaching and learning from 
grade one onwards. For science instruction in grade four, the national curricula in 
Albania, Kosovo, Montenegro, and North Macedonia contain statements about the 
use of digital devices, such as computers, tablets, or calculators. For example, in 
Montenegro, students are encouraged to become familiar with using digital devices 
in all subjects. Teachers may decide to use computers in the classroom to demonstrate 
experiments or to get students to practice their skills, although this is not mandatory 
(Kelly et al., 2020). 

There are some special initiatives related to ICT in the region. In Croatia, the 
development of digital competencies is seen as a cross-curricular endeavor and, as 
part of the reforms that started on an experimental basis in the school year 2018/2019, 
school equipment is being upgraded and ICT slowly introduced into teaching prac- 
tices. Before this reform, computers were rarely used outside of informatics lessons 
in primary education, and the infrastructure and conditions for ICT use in teaching 
varied considerably between schools and counties. ICT use is considered central to 
improving and modernizing teaching and learning in Croatia (Elezović & Muraja, 
2020). Across the wider region, there are several other ICT initiatives, including the 
twenty-first century schools program managed by the British Council; this provides 
support to around 4500 primary schools in the region, to strengthen digital education 
and digital literacy (British Council, 2020). 

According to a European Council recommendation (European Union, 2018), 
digital competence can be defined as “the confident, critical and responsible use 
of, and engagement with, digital technologies for learning, for work, and for partic- 
ipation in society." This definition, accompanied by a comprehensive framework 
known as DigComp 2.1 (Carretero et al., 2017), has become a popular reference tool 
for many European countries in incorporating digital competence in their national 
curricula, and has also been adopted in the Dinaric region. Bosnia and Herzegovina 
refers to the European definition of digital competence for school education in its 
curricula and related strategy documents, whereas Albania and Serbia refer to the 
European definition in addition to their own national definition. In contrast, Croatia 
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has its individual national definition of digital competence, and Montenegro and 
North Macedonia do not have a common definition. In principle, digital compe- 
tence in primary and secondary education can be implemented as a cross-curricular 
theme, as a separate subject, or integrated into other subjects. In the school year 
of 2018/2019, the national curriculum for primary education (ISCED 1) in Serbia 
included digital competence as a cross-curricular component, while in Montenegro 
and North Macedonia it was included as a compulsory separate subject (European 
Commission/EACEA/Eurydice, 2019b). 


4 The Role of Assessments for Quality Assurance 


4.1 Decision-Making Authorities and Distribution 
of Responsibilities 


Throughout the Dinaric region, the decision-making authority and responsibilities 
related to different factors within the education system are generally structured in 
a similar way, reflecting the similar circumstances across the region when the end 
of communism around 1990 resulted in the process of transition from centralized 
planning to a market-based economy. The general trend of recent years has been to 
delegate more responsibility from central governmental levels downwards through 
the hierarchies within the systems. For example, North Macedonia has transferred 
greater responsibility for educational matters to municipalities and, since 2005, the 
municipalities have been responsible for funding andrunning pre-school institutions, 
primary schools, and secondary schools. They were also empowered to open new 
establishments, distribute central funding, maintain and audit schools, and appoint 
principals, teachers, and school board members. These reforms have led to rela- 
tively high levels of school autonomy in terms of resource management and student 
assessment policies. Principals have increased responsibility when it comes to human 
resource activities related to teachers, and teachers have more control over the devel- 
opment of classroom assessments. The North Macedonian government considered 
this step as an important move toward improving the quality of education, equality of 
opportunities, and the overall efficiency of the system (Kitchen et al., 2019; Lameva, 
2020). 

Albania has gone through a similar series of educational reforms, deemed neces- 
sary to improve educational outcomes such as student performance. In 2003, regional 
education directorates and offices were set up to support the Ministry of Educa- 
tion, Sport, and Youth with implementing the national education policies in schools. 
Further decentralization of school services in 2019 made regional directorates respon- 
sible for school evaluation. Reforms led to increased school autonomy, which should 
enable schools to make effective use of resources and reflect on their policies and 
practices. As one example, after approval from the local educational institutions, 
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schools are now entitled to draft individual curricula based on the ministry-approved 
curriculum framework (Maghnouj et al., 2020). 

In Kosovo, municipalities are in charge of the construction and maintenance of 
educational facilities and for ensuring a healthy environment for staff and students. 
They are also responsible for enrolling students and hiring teachers and other school 
staff (DQASAL/MEST, 2020). 

In Serbia, the Ministry of Education, Science, and Technological Development 
is responsible for educational research, planning, development, supervision of pre- 
school, elementary, secondary, and higher education, and the organization, evalua- 
tion, and supervision of employees in the education sector. Public educational insti- 
tutions are financed mainly from national and local budgets. It has been estimated 
that, during the last few years, almost 90% of the total funds have been provided 
from the national budget, with the remainder derived from local budgets. Recently, 
municipalities have become more involved in the funding of education; however, 
the government still provides the essential materials and technical resources that 
are a necessary minimum for school operations (Derié et al., 2020). The process of 
democratization, decentralization, and depoliticization of education in Serbia started 
in 2000 at all levels of education, targeted at educational policy and practice and 
quality and equity in education (Spasenovié et al., 2007). 


4.2 Quality Assurance Structure in Education Systems 


Quality assurance is an integral part of education systems, impacting educational 
outcomes. Exactly how quality assurance is implemented and where it is located 
within the education system depends on the structure of a system. It can form part 
of the role of the ministry of education or be organized as a completely independent 
external institution. Across the Dinaric region, multiple relevant institutions deal 
with quality assurance in a pre-university context. These include internal quality 
assurance processes, which can be defined as a continuous process of rethinking 
and reevaluating current practice, guided by various procedures, including statis- 
tics, indicators, and other information collected from and serving different stake- 
holders. Conversely, external quality assurance processes may include national and 
international assessments and evaluations by school inspectorates. 

The education systems in the Dinaric region usually subdivide quality assurance 
into the following sections: early childhood education and school education, higher 
education, and adult education and training. Quality assurance in the school sector 
is structured in different ways and focuses on varying aspects throughout the region. 

In Albania, the bodies responsible for quality assurance in the school sector are the 
National Agency for Education, Training, and Qualifications and the State Education 


26 P. Koršñáková and S. Dohr 


Inspectorate. Internal quality assurance mechanisms in Albania include school self- 
evaluation, which is conducted via the Albanian school performance charter (Alba- 
nian Ministry of Education and Sports, 2014), a document that incorporates infor- 
mation on achievement, support for the school in performing their duties, and stan- 
dard indicators, and functions as a self-evaluation instrument for schools. National 
student assessments have been an established external quality assurance instrument 
since 2016, conducted at the end of grades five, nine, and 12, and managed by the 
Centre for Educational Services (European Commission/EACEA/Eurydice, 2020). 

Quality assurance in Bosnia and Herzegovina is integrated into all institutions 
related to education, and the Agency for Pre-primary, Primary, and Secondary Educa- 
tion (APOSO) holds the authority to establish learning standards, evaluation of 
achievements, and development of the common core curriculum (Bosnia and Herze- 
govina Council of Ministers, 2018). At school level, internal evaluation is part of 
the governance structure and external evaluation is carried out by expert advisors, 
who may review school operations or the work of teachers (European Commis- 
sion/EACEA/Eurydice, 2020). The first external evaluation at state level concerning 
grade four was carried out as a national assessment in 2002 by a predecessor of 
APOSO. National assessments in Bosnia and Herzegovina do not occur regularly 
due to a lack of financial resources (Popić & Džumhur, 2020). 

Quality assurance in Croatia is managed by several governmental agencies with a 
focus on the specific levels of education. The Teacher Training and Education Agency 
is mainly responsible for quality assurance of the pre-primary, primary, and secondary 
education sector (European Commission/EACEA/Eurydice, 2020). External national 
examinations are usually conducted by the National Center for External Evaluation 
of Education (NCEEE); since its establishment in 2006, national exams have been 
conducted at rather irregular intervals and are seen as diagnostic tools. In 2011, IEA 
studies were introduced in the form of a combined TIMSS and PIRLS (Progress in 
International Reading Literacy Study; see IEA, 2021) assessment at grade four. Since 
then, these IEA studies have become an independent international tool for deepening 
the understanding of student achievement and student attitudes in primary education 
(Elezovié & Muraja, 2020). 

In Kosovo, the Ministry of Education Science, and Technology (MEST) is 
primarily responsible for ensuring the quality of pre-university education. Quality 
assurance activities are carried out by various subdivisions and subordinate agen- 
cies of the ministry, such as the Education Inspectorate or the Division for Quality 
Assurance, Standards, Assessment, and Licensing (DQASAL). The latter was estab- 
lished in 2001 to develop education policies and to conduct national and international 
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assessments. Among other tasks, they also evaluate the quality of service of educa- 
tional institutions or advise executives of education institutions. TIMSS 2019 repre- 
sents the first international assessment in Kosovo dedicated to grade four students2 
(DQASAL/MEST, 2020). 

The quality assurance system in Montenegro stipulates that all educational insti- 
tutions carry out annual quality assurance and improvement activities in the form 
of self-evaluation. In addition, the Bureau for Education Services is responsible for 
external quality assurance in the sector of pre-school, primary, and secondary educa- 
tion. All quality assurance activities in Montenegro are based on the Rulebook on the 
content, form, and manner of quality assessment of educational work at institutions 
(Government of Montenegro, 2020). This covers areas such as students’ achievement, 
quality of teaching and learning, the governance and management of the institution, 
or cooperation with parents (European Commission/EACEA/Eurydice, 2020). 

Quality assurance in the pre-school, primary, and secondary education sector in 
North Macedonia is carried out by the State Education Inspection (SED, the Bureau 
for Development of Education (BDE), and the State Examination Center (SEC). 
The last body is responsible for the external evaluation of students” achievement 
and teaching (European Commission/EACEA/Eurydice, 2020). The first national 
assessment of mathematics in North Macedonia was organized at the end of grade 
four in 2000 and followed by an assessment for natural sciences in 2006, which 
aimed to provide the relevant education institutions with valid data about student 
attainment that could be used to create educational policy and provide feedback for 
schools and teachers. Over the period from 2014 to 2016, electronic external testing 
was carried out for students in grade four. A new law on primary education demanding 
national assessment is currently being developed, with the first tests planned for 2021 
(Lameva, 2020). 

The Serbian Ministry of Education, Science, and Technological Development 
supervises external quality assurance activities that are conducted by the National 
Education Council, the Institute for the Improvement of Education, and the Insti- 
tute for Education Quality and Evaluation. These institutions are responsible for 
determining educational standards, national testing, suggesting improvements, or 
adapting educational policies. The main aim of the internal quality assurance lies in 
strengthening school self-governance and autonomy and in ensuring the professional 
development of teachers by identifying ways for improvement of their practices and 
management (European Commission/EACEA/Eurydice, 2020). The first national 
testing of grade four students was undertaken in 2006 by the Institute for Educational 
Quality and Evaluation. In 2015, Serbia introduced annual tests in mathematics for 
students in grades four and six. 


2 It is important to mention the existing parallel education system, which has separated Albanian 
Kosovars from Serbian Kosovars since the 1990s (OECD [Organisation for Economic Cooperation 
and Development], 2006; Gabršček & Dimc 2000). Data collected in TIMSS 2019 exclusively 
covers the Albanian Kosovars. 
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43 TIMSS 2019 Implementing Institutions 


International large-scale assessments have become an integral part of external quality 
assurance across the Dinaric region in recent years, as internationally standardized 
testing provides high-quality data and potentially beneficial comparable insights into 
the strengths and weaknesses of education systems. Although TIMSS had been previ- 
ously implemented in many parts of the region, 2019 was the first time that all educa- 
tion systems in the region chose to use the assessment to monitor the mathematics and 
science achievement of their grade four students. The national institutions involved 
in implementing TIMSS 2019 across the region have been vital for the success of 
the study in the participating education systems. 

The Educational Services Center (QSHA) in Albania was established in 2017 as an 
evolution from multiple older institutions. Currently, QSHA is a special institution for 
organizing, monitoring, and supervising educational reforms and quality assurance 
in the field of evaluation of achievement and exams. It exercises the technical, profes- 
sional, supportive, and recommendatory function of the assessment and organization 
policies of state exams (e.g., the school-leaving examination). The institution is also 
involved in all international large-scale assessments in which Albania participates 
(European Commission/EACEA/Eurydice, 2020). 

In Bosnia and Herzegovina, APOSO was established to serve as an independent 
administrative organization promoting quality of education at pre-primary, primary, 
and secondary levels. The agency was also appointed to conduct the TIMSS admin- 
istration in Bosnia and Herzegovina (Popić & Džumhur, 2020). The agency has its 
seat in Mostar and two regional units located in Sarajevo and Banja Luka. Plan- 
ning and implementing international comparative studies form part of the work of 
the regional unit in Sarajevo, together with other tasks related to learning standards, 
student achievements, and evaluating the results in primary and secondary education. 
APOSO is further involved in developmental and research programs and projects, 
designing and maintaining relevant databases, performing psychometric measure- 
ments, statistical analysis, publishing activities, and performing translation tasks 
(APOSO, 2020). 

The NCEEE in Croatia originated with the mission to provide valid and objective 
monitoring and external evaluation of the national education system, in collaboration 
with its stakeholders, to define and improve the quality of education. The NCEEE, 
located in Zagreb, is responsible for comprehensive external evaluation of the pre- 
tertiary education system in Croatia. In addition to the objective and transparent 
external evaluation of learning outcomes and the development, implementation of the 
system of external evaluation of education, and conduct of international research in 
education, the NCEEE licenses, certifies and educates key stakeholders in education 
on the quality of education at all levels of the system, and research and development 
in educational measurement, and continually promotes and develops the quality of 
the education system to foster positive change (NCEEE, 2020). 


3 School-leaving examinations are commonly referred to as "Matura" across the region. However, 
preference was given to the term “school-leaving examination" throughout the chapter. 
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In Kosovo, DQASAL was first established in 2001 as part of the Ministry of Educa- 
tion, Science and Technology of the Republic of Kosovo, and is responsible for the 
development of educational policies and national and international student assess- 
ments. The competencies of the division include the organization and implementation 
of international assessments, the dissemination of study results, and the implemen- 
tation of the national school-leaving examination. The institution further consults 
and cooperates with municipal education institutions to support them with self- 
assessment and development based on test results, and organizes teacher seminars 
about assessing student achievement (DQASAL/MEST, 2020). 

The government of Montenegro established the Examination Center in Podgorica 
in 2005. The Examination Center conducts the external evaluation of achieved stan- 
dards of knowledge and competences of students, organizes counseling for teachers 
to provide professional assistance for a better implementation of external knowl- 
edge assessment, prepares and organizes regional and national competitions for 
primary and secondary school students, organizes and conducts state examinations 
(e.g., the school-leaving examination), investigates and develops external assess- 
ments, and assesses the international comparability of the quality of the education 
system, including implementing relevant international assessments (Examination 
Center, 2005). 

The North Macedonian State Examinations Center (SEC) was established in 2009 
with headquarters in Skopje as an independent successor of the Assessment Unit. Its 
role is to evaluate the quality of education through external assessments. SEC's 
competencies include the organization, implementation, and support of external 
assessments, and evaluating student achievement in primary and secondary education 
in North Macedonia. SEC also prepares and implements national external assess- 
ments, such as the school-leaving examination at the end of secondary school. The 
responsibilities of the center additionally include preparing and monitoring examina- 
tion materials for high school and secondary vocational education. As an institution, 
SEC oversees the training and licensing of principals of primary and secondary 
schools and also issues certificates (licenses) to teachers and professional associates 
from primary and secondary education. SEC is responsible for the preparation and 
implementation of international assessments (State Examinations Center, 2020). 

The Institute for Educational Research (IER) is an independent research institute 
in Serbia, founded in 1959. Granted the status of a scientific institution in 1961, 
the institute is currently a leading research institution in the field of education and 
contributes to the development and improvement of the quality of education in Serbia. 
Its main objectives are researching excellence and innovation, and promoting science 
and the awareness of the role of educational research. It integrates basic, applied, and 
developmental multidisciplinary research to contribute to scientific, educational, and 
social development in Serbia. IER was responsible for implementing TIMSS from 
2003 to 2019, focusing on grade four since 2011 (IER, 2012). 
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5 TIMSS Survey Design and Its Implementation 
in the Region 


The TIMSS 2019 assessment was organized around a content dimension and a 
cognitive dimension, both of which consisted of several sub-domains. The assess- 
ment framework describes the content domains (e.g., measurement and geometry 
in mathematics or life science in science) and topics within these domains include 
specific abilities that grade four students should be able to demonstrate. The cognitive 
domains of knowing, applying, and reasoning describe the thinking skills students 
are expected to use to solve the mathematics and science problems. Each item in the 
assessment was assigned to a specific content and cognitive domain (Centurino & 
Jones, 2017; Lindquist et al., 2017) and, similarly, the aspects of the learning context 
covered by TIMSS 2019 were addressed in the context questionnaires framework 
(Hooper et al., 2017). 

In recognition of the increasing use of information technology for learning and 
assessment in recent years, the TIMSS pen-and-paper assessment transitioned to 
a digital data collection mode in 2019 (“eTIMSS”; see Mullis & Martin, 2017), 
however the pen-and-paper option was retained as an option. 

Among the Dinaric region TIMSS 2019 participants, only Croatia opted for 
eTIMSS. The eTIMSS design was more extensive because it also included four blocks 
of problem-solving and inquiry (PSI) tasks and items. However, for this research we 
only considered the item blocks in eTIMSS 2019 that had a matching counterpart 
in the pen-and-paper format, although they were adapted to make appropriate use of 
digital components (e.g., slightly modified so students could make use of additional 
“drag and drop,” or “sorting” features). 

The individual grade four student response burden for the TIMSS 2019 assess- 
ment was the same as it has been since TIMSS 2007, allowing 72 min for the assess- 
ment and 30 min for the student questionnaire, with a short break before under- 
taking the second part of the assigned student achievement booklet and again before 
completing the student questionnaire. TIMSS’ ambitious reporting goals require 
many more questions in the assessment than could be answered by a single student 
in the amount of testing time available. TIMSS thus uses a matrix sampling approach 
that involves packaging the entire assessment pool of mathematics and science items 
at each grade level into a set of 14 student achievement booklets, with each student 
completing just one booklet. Item response theory scaling methods are then used to 
assemble a comprehensive picture of the achievement of the entire student population 
of a country from the combined responses of individual students to the assessment 
booklets that they are assigned (Martin et al., 2017). 

Participants of TIMSS 2019 administering the assessment at grade four could 
choose to use booklets with some less difficult blocks in mathematics than the regular 
TIMSS grade four assessment. Out of the seven TIMSS 2019 participants in the 
Dinaric region, all but Croatia and Serbia opted for the less difficult mathematics 
assessment. The less difficult mathematics assessment was designed for students 
that are still developing fundamental mathematics skills. The results of these tests 
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are reported on the same achievement scale as other TIMSS participants, but the less 
difficult items extend the TIMSS mathematics achievement scale to provide better 
measurement at the lower end of the scale. Experiences with TIMSS Numeracy in 
2015 and PIRLS Literacy in 2016 (the less difficult version of IEA’s PIRLS reading 
assessment) indicate that lower-performing students are more strongly motivated by 
less difficult items, and better demonstrate what they know and can do, resulting in 
fewer omitted items and higher completion rates. Consequently, the results from the 
less difficult TIMSS and regular TIMSS are comparable, regardless of the version 
of the assessment the students have taken (Mullis & Martin, 2017). 


5.1 Sampling and Some Key Characteristics of the Target 
Grade 


The international TIMSS sample design calls for a minimum of 150 schools with one 
or more intact classes in grade four, resulting in a student sample of approximately 
4000 students per participating entity. The target grade for this publication is grade 
four and the population is defined internationally as “the grade that represents four 
years of schooling, counting from the first year of ISCED Level 1” (Martin et al., 
2017, p. 81). ISCED Level 1 corresponds to primary education or the first stage 
of basic education. In addition to the grade, TIMSS attempts to avoid assessing 
very young students and sets the minimum average age at the time of testing of 
9.5 years. If this condition is not met, participants need to assess the next higher 
grade. For the participants in this publication, TIMSS 2019 was administered at 
grade four. The average student’s age on the day of testing in the region ranged from 
9.8 in Montenegro and North Macedonia to 10.6 in Serbia. Data from 3270 to 5628 
students per participating education system were collected between 11 March and 15 
May 2019 across the region. The student assessment was administered in the main 
languages of instruction within the education system. North Macedonia administered 
the test in one minority language, namely Albanian, and Croatia prepared the home 
and school questionnaire in Serbian and Italian as well as Croatian (see Table 2). 
TIMSS employs a two-stage random sampling design. A sample of schools is 
drawn first, then one or more intact classes of students are selected from each of 
the sampled schools. Intact classes of students are sampled rather than individ- 
uals from across the grade level or of a certain age because TIMSS pays partic- 
ular attention to students’ curricular and instructional experiences in the classrooms. 
Sampling complete classes also has the operational advantage of less disruption to 
the school day compared to individual student sampling (Joncas & Foy, 2012). Each 
TIMSS participant needed to define the national target population and apply the 
TIMSS sampling methods to achieve a nationally representative sample of schools 
and students. The development and implementation of the national sampling plan 
is a collaborative exercise involving the country’s national research coordinator 
(NRC) and the TIMSS 2019 sampling experts. This procedure ensures that the school 
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sampling frame (the school population list from which the school sample is drawn) 
provided by the NRC is complete, checking that categories of excluded students are 
clearly defined, justified, and kept to a minimum. The objective is to draw a nationally 
representative sample of students, while making sure both international and national 
requirements regarding sampling precision are met. National requirements are often 
addressed by applying a specific stratification approach (Meinck, 2020). 

School location within the participating education systems and whether this loca- 
tion was urban or rural were the major explicit stratification variables used across 
the region. Albania also stratified their schools by school type (whether public or 
private), Kosovo by school shifts, and North Macedonia and Bosnia and Herzegovina 
by language of instruction. TIMSS aims to cover the whole grade four student popu- 
lation in all countries, but allows the exclusion of specific populations for practical 
reasons. At the within-school level, across the Dinaric region, classes or students with 
functional or intellectual disabilities were excluded, as were non-native language 
speakers, however, there were some differences at the school level. Most of the 
exclusions could be attributed to a small number of grade four students in schools 
(less than three), and the exclusion of schools that only taught students with special 
needs or provided instruction in a minority language, or those studying a different 
curriculum (e.g., international schools). In Albania, Montenegro, and Serbia, all 
students who were not taught in the majority language of instruction were speci- 
fied as language-based exclusions. In TIMSS 2019, Kosovo and Serbia exceeded 
the five percent maximum exclusion rate that is usually set for studies designed for 
cross-national comparisons. In both cases, this happened because of their linguis- 
tically diverse populations; TIMSS 2019 could not always be administered in all 
recognized languages of instruction. In Kosovo, schools with the Bosnian or Serbian 
language as the primary language of instruction accounted for 8.56% of the overall 
exclusion rate, and the schools serving the minority populations in Serbia accounted 
for 8.21%. The analyses in this study only used the data from assessments delivered 
in the prevailing Albanian language in Kosovo and the Serbian language in Serbia. It 
is important to underline that the TIMSS 2019 test was administered in the language 
of instruction. The vast majority of the sampled students responded to the test in 
a language that they were using at home (Table 3). However, the data collected by 
TIMSS also indicates that classrooms are not linguistically homogenous. 

After sampling had been completed and all data collected, the TIMSS sampling 
experts documented population coverage, and school and student participation rates, 
and constructed appropriate sampling weights for data analysis. The target for TIMSS 
is asampling participation of 100% for all sampled schools, classrooms, and students, 
and the achievement data are reported according to this target. TIMSS participants 
were assigned to one of three categories on the basis of their sampling participation: 
category | (considered to have met all TIMSS 2019 sampling requirements and to 
have acceptable participation rates), category 2 (meeting the participation require- 
ments only after including replacement schools), and category 3 (failed to meet the 
participation requirements even with the use of replacement schools) (LaRoche & 
Foy, 2016). Across the region, all participants fell in category 1, indicating that the 
quality and validity of the results can be trusted. 
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Table 3 How often do students speak the language of the test at home? 


Education system Percentage of students (%) 
Always Almost always | Sometimes | Never 

Albania 86.58 6.30 6.21 0.91 
Bosnia & Herzegovina | 84.38 8.34 6.50 0.79 
Croatia 68.19 20.04 10.48 1.29 
Kosovo 87.16 5.97 6.44 0.43 
Montenegro 71.27 11.74 10.81 6.17 
North Macedonia 79.10 7.91 10.00 2.99 
Serbia 84.77 9:55 5.12 0.56 


Source Mullis et al., (2020) 


5.2 Quality Assurance for the TIMSS 2019 Test 
Administration Across the Dinaric Region 


Strict quality assurance procedures ensure large-scale assessments produce high 
quality, internationally comparable data. In TIMSS 2019, quality assurance was 
an integral part of the study and was implemented at both the international and 
national levels. This encompassed all main activities from the assessment frame- 
work, including assessment and questionnaire development, sampling, instrument 
preparation, data collection, scaling, and data analysis. The TIMSS quality assur- 
ance components that were implemented during the data collection period were the 
international and national quality control programs, and the TIMSS Survey activities 
questionnaire (SAQ), completed by all NRCs. 

The SAQ is used by NRCs to document their experiences of the TIMSS 2019 
main data collection processes. The questions cover all activities from school 
sampling, preparing the national study instruments including their translations, 
reviews, printing, checking and distribution, selecting and training the school coor- 
dinators and test administrators, maintaining the security of the assessment mate- 
rials and the confidentiality of the responses, observing testing sessions, scoring the 
obtained responses, and performing data entry and checks. All these activities are 
additionally described in the TIMSS 2019 Survey operations procedures units, which 
are documents designed to provide guidance to NRCs. The NRCs reported that these 
guidance documents were considered clear and useful, although some participating 
entities encountered various challenges during the preparation of the assessment 
instruments (e.g., difficulties in using the Adobe InDesign( package to prepare the 
national versions of the international instruments). 

Three interlinked aspects were related to data availability and quality: (1) school 
coordinator and test administration appointment and training; (2) the implementa- 
tion of the national quality control program within the participating schools; and 
(3) independent observations undertaken as part of the international quality control 
program, which was overseen by the TIMSS & PIRLS International Study Center. 


34 P. Koršñáková and S. Dohr 


All school coordinators from the Dinaric region were appointed by the partici- 
pating schools, and all undertook formal training. In most cases, these school coor- 
dinators trained the test administrators in the participating schools. However, in 
Bosnia and Herzegovina, most of the test administrators were trained by staff from 
the national study center. In Kosovo and North Macedonia, the test administrators 
were external to the participating schools. They brought the assessment materials for 
the testing session to the school, conducted the session, and collected all materials 
after the session was finished. Based on the completed TIMSS test administration 
forms, the most frequent problem for students was encountering unknown words 
and/or tasks that they had not yet covered in school (e.g., fractions in Croatia). 

Aspart of the national quality control programs, the data collection was observed 
in the participating schools; these observations took place in all participating entities 
in TIMSS 2019 from the Dinaric region, except Kosovo and Montenegro, who had 
insufficient budget for this activity. Testing sessions in the recommended 10% of 
participating schools were observed in Albania, Croatia, and Serbia, 12% of the 
sampled schools were visited in Bosnia and Herzegovina, and 100% of schools were 
visited in North Macedonia. The national quality control monitors (NQCMs) did 
not document any major problems or special circumstances that occurred repeatedly 
during the administration of the TIMSS assessment. 

In addition to the information from national quality control and the SAQ, interna- 
tional quality control monitoring was also part of TIMSS 2019. International quality 
control depended on reported observations from selected experts appointed as inter- 
national quality control monitors (IQCMs). These individuals were employed inde- 
pendent of the TIMSS 2019 national centers and personally trained for monitoring 
activities by the TIMSS & PIRLS International Study Center (see Johansone & 
Flicop, 2020 for more detailed information on these aspects). 
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Abstract IEA’s Trends in International Mathematics and Science Study (TIMSS) 
uses the curriculum as the major organizing concept in considering how educational 
opportunities are provided to students. “Opportunity to learn” is generally defined 
by the instructional time spent on a specific subject area and instructional content. 
TIMSS data can be used to analyze key aspects surrounding students’ opportunities to 
learn mathematics and science, in combination with background factors that influence 
how students use these opportunities. The results concerning opportunity to learn can 
be compared at different levels, related to the prescribed curriculum, the implemented 
curriculum, and attained educational goals. Across the Dinaric region, the TIMSS 
2019 data showed that there were some discrepancies between intended, imple- 
mented, and attained curricula. Officially prescribed contents in national curricula 
differed from the teacher reports of content taught in school. The analyses also 
revealed that there were no significant common relations between the percentage 
of students that were taught the topics and mean national achievement in TIMSS 
2019 across the Dinaric region and that, contrary to expectations, school content 
coverage could not solely explain observed student achievement. Other important 
elements may have a mediating effect, such as the quality of instruction or time and 
opportunities available for learning outside school. 
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1 Introduction 


Opportunity to learn (OTL) can be considered the “observable structure” of teaching 
(Schmidt & Maier, 2009; Schmidt & McKnight, 1995) and thus makes a valuable 
contribution toward learning outcomes. OTL connects nationally prescribed contents 
and methods of teaching and learning (the intended curriculum) with learning 
contents, class environment, and school climate (the implemented curriculum), and 
students’ outcomes (the attained curriculum). 

Alignment between educational goals, intended and implemented curricula, and 
educational outcomes is deemed an essential characteristic of effective education. 
The expectation is that better alignment between these leads to more effective educa- 
tion and hence better student performance. The concept of OTL is commonly used 
to compare content covered, as part of the implemented curriculum, with student 
achievement. As such, OTL can be viewed as a facet of the broader concept of 
alignment (Scheerens, 2017). 

Accordingly, OTL can be characterized as the alignment between teaching 
processes and student achievement, or as “the alignment of standards and output 
measures, mediated by teaching processes” (Scheerens, 2017, p. 41), it can also refer 
to a wide range of inputs and processes within a school context that support intended 
student outcomes. In doing so, instructional time and content have also consistently 
been characterized as core elements of OTL, along with a number of instructional 
quality indicators. Carroll (1963) was first to define OTL as “the amount of time 
allowed for learning, for example by a school schedule or program" (see Carroll, 
1989, p. 26). The central concept is the notion that students cannot learn content that 
has not been presented in class. Carroll (1989) included OTL as one of five variables 
in a formula that he used to express a student's degree of learning. Some of these 
variables can be measured to a certain extent by time, in terms of the amount of time 
a student needs to be given to learn, understand, and master a task, and the amount of 
time allowed for this learning in the curriculum. This has led researchers to examine 
how time dedicated to instruction (the instructional time) may be related to student 
achievement. According to Kurz (2011, see also Elliott & Bartlett, 2016), Stevens 
(1996) distilled the first comprehensive conceptual framework of OTL, bringing 
together four elements: content coverage, content exposure (time on task), content 
emphasis (the emphasis on the cognitive processes required), and quality of instruc- 
tional delivery (the methods and quality of the instructional practices used to deliver 
the content). 

Content exposure refers to the estimated total amount of time actually devoted 
to covering the specific content (Leinhardt & Seewald, 1981). Terms like instruc- 
tional time, amount of time devoted to teaching certain subject areas, and amount of 
teaching periods (or hours per week or year) are traditional measures of this particular 
dimension of OTL (Stedman, 1994; Wang, 1998). Brophy (2000) found that more 
time allocated to teaching specific content in classrooms positively contributed to 
student achievement. To provide students with the necessary opportunities to learn 
the intended curriculum, teachers must allocate instructional time toward addressing 
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specifically prescribed teaching goals to achieve the requisite student outcomes 
(Elliott & Bartlett, 2016). Such measures may be categorized by teacher-oriented 
indicators, such as allocated time (the time scheduled for instruction), or by more 
explicitly student-oriented indicators, such as instructional time (proportion of allo- 
cated time used for instruction), engaged time (proportion of instructional time during 
which students are engaged in learning), and academic learning time (proportion of 
engaged time during which students are experiencing a high success rate of learning). 
Researchers have found time-based OTL indices to be moderately related to student 
achievement after controlling for student ability and socioeconomic status (Elliott & 
Bartlett, 2016, p. 5). 

Teachers must also ensure that they cover the content outlined by the formal 
curricula, as these are the topics that students are likely to be assessed on. In 1964, 
IEA undertook the First International Mathematics Study (FIMS) in twelve countries, 
to investigate the outcomes of various school systems for mathematics, which was, 
at the time, undergoing reforms in many educational systems (Husén, 1967). This 
results from this early IEA study led to increased research interest in content-based 
conceptualizations of OTL, designed to evaluate the content overlap between enacted 
and assessed curricula (Elliott & Bartlett, 2016). Anderson (1986, p. 3682) noted that 
the “opportunity to learn from the Husén perspective is best understood as the match 
between what is taught and what is tested.” Perhaps the most important measure of 
content coverage in current policy efforts is the alignment of teachers’ instruction 
with state standards and/or assessments (Scheerens, 2017). 

Substantial empirical evidence has documented the importance of OTL variables 
in explaining students’ test scores and found that students were more likely to respond 
to an item correctly if they had the opportunity to learn the tested concepts and skills, 
especially if students had this opportunity during the year the assessment was deliv- 
ered. Wang (1998) examined the impact of Stevens’ (1996) four dimensions of OTL 
on outcomes and found that OTL was a significant predictor of student achievement 
in both written tests and other scores students received in schools. Variations in the 
effects of OTL could be attributed to differences in test format. In their study of 
English language learner achievement, Aguirre-Mufioz and Boscardin (2008) stated 
that content exposure was the most significant predictor of students’ written test 
scores, whereas the quality of instructional delivery was the most significant predictor 
of practical test scores. 

IEA’s Trends in International Mathematics and Science Study (TIMSS) is one 
of the few international studies that are curriculum based, and it thus routinely 
collects information about OTL at different levels. As stated in the TIMSS frame- 
work, “TIMSS uses the curriculum, broadly defined, as the major organizing concept 
in considering how educational opportunities are provided to students and the factors 
that influence how students use these opportunities” (Mullis, 2017, p. 4). 

Along with total instructional hours per year, TIMSS collects information, on a 
national level, on whether the curriculum or any other official document prescribes 
the percentage of total instructional time to be devoted to mathematics and science 
instruction at the grade four. 
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1.1 Framework and Research Questions 


For our research, we considered a restricted concept of OTL. We used a conceptual 
framework that combined elements from educational effectiveness, a curriculum 
model, and the notion of OTL to analyze curriculum effects. In our model the 
intended curriculum is defined as the curriculum that an education system intends 
to implement, as stated in their official policy documents. At the school, classroom, 
and teacher level are the actors that actually implement that curriculum. Finally, the 
students (hopefully) attain the content taught. At each of these levels, we can observe 
specific curriculum factors that together define the whole curriculum (Bokhove et al., 
2019). 

We use the term opportunity to learn specifically with reference to the mathematics 
and science topics covered in classroom instruction. This reflects both the narrow 
curricular sense in which the concept was originally developed by Carroll (1963) and 
in the studies implemented by IEA. We chose to focus on the definition of OTL as 
time of exposure and quality of content for two reasons: (1) the provision of content 
is the fundamental rationale of schooling and the education system, and (2) this is an 
aspect of schooling that both reflects education policy and is amenable to education 
policy reform (see Scheerens, 2017, p. 41). 

Seven participants from the Dinaric region took part in TIMSS 2019, namely 
Albania, Bosnia and Herzegovina, Croatia, Kosovo, ! Montenegro, North Macedonia, 
and Serbia. Our analysis of the TIMSS 2019 data was designed to address two key 
research questions: 


(1) What can TIMSS 2019 tell us about students’ opportunities to learn mathe- 
matics and science across the Dinaric region? 

(2) Can students' opportunities to learn be related to their mathematics and 
science achievement? 


We explored key aspects of OTL included in the TIMSS international frame- 
work at grade four. In TIMSS, content exposure is defined as the instructional time 
prescribed and devoted to mathematics and science curricula at the grade four, and 
content coverage is defined as the number and content of mathematics and science 
topics intended to be taught and effectively taught in classes (Mullis & Martin, 2017). 
We explored both content exposure and content coverage at the level of intended 
curriculum and at the level of implemented curriculum. Our analyses conceptu- 
alized OTL as the overlap between intended content coverage and implemented 
content coverage; OTL is thus a measure of the relation between implemented content 
and intended content. Content coverage addresses the degree to which content was 
covered throughout the continuation of the school, in order to see if students have 
adequate opportunity to learn topics assessed by the test. Content exposure refers to 
the total amount of time spent on covering the specific subjects. Finally, we assessed 


! This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 
(United Nations 1999) and the International Court of Justice (ICJ) Opinion on the Kosovo declaration 
of independence (ICJ 2010). 
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the relationship between OTL and the attained curriculum, namely the mathematics 
and science achievement of TIMSS grade four students across the Dinaric region. 


2 Methods and Data 


As part of TIMSS, data about the contexts for learning are collected through ques- 
tionnaires completed by students and their parents, teachers, and school principals 
(for more information, see TIMSS & PIRLS International Study Center, 2018). All 
the data used in our analyses were collected by the TIMSS 2019 grade four assess- 
ment. We carefully selected relevant data collected by the TIMSS 2019 background 
questionnaires to inform our analyses using variables at all three curriculum levels 
(Table 1). 

When examining the data available from TIMSS 2019 for grade four students, 
we limited our observations to two basic elements of the concept of opportunity to 
learn: time exposure (namely the time allocated to instruction in a topic in school) 
and content exposure (the amount of content presented to students by teachers). To 
observe the effect of OTL on final educational outcomes, we related the national aver- 
ages for these elements to the national mean achievement of students in each topic, 
which was measured independently by TIMSS 2019 in all participating education 
systems. 

Mathematics and science curricula were assessed using broad content domains: 
for mathematics, these were number, measurement and geometry, and data, and, for 
science, these were life science, physical science, and earth science. Each content 
was divided into specific topics, numbering 17 in total for mathematics (seven for 
number, seven for measurement and geometry, and three for data; Lindquist et al., 
2017) and 26 in total for science (seven for life science, twelve for physical science, 
and seven for earth science; Centurino & Jones, 2017). These topics served as a basis 
for international development of items for students to solve in the TIMSS assessment 
(Mullis & Martin, 2017). Therefore they represent the content assessed by TIMSS. 

In TIMSS, information on mathematics and science contents covered in national 
curricula up to grade four were collected at a system level by a curriculum question- 
naire completed by the national research coordinator (NRC) in each participating 
entity from the Dinaric region. Such information reflects the content of the intended 
curricula. We assessed the implemented curricula using international data on prin- 
cipals’ and teachers’ responses to the TIMSS context questionnaires? (TIMSS & 
PIRLS International Study Center, 2018). The attained curriculum was evaluated 
using students’ mathematics and science achievement scores in the TIMSS 2019 


2 In TIMSS teachers teaching mathematics and science to the sampled class are selected to respond 
to the teacher questionnaire. As they do not represent the teacher population in each system, teacher 
data is analyzed as an attribute of students. 
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assessment.” In addition, to analyze the achieved curriculum in more detail, we subdi- 
vided student achievement according to the specific mathematics (number, measure- 
ment and geometry, and data) and science (life science, physical science and earth 
science) content domains. The teacher questionnaire (see TIMSS & PIRLS Interna- 
tional Study Center, 2018) also asked teachers to report which topics they had already 
taught to students participating in TIMSS prior to the assessment. These reports of the 
implemented curriculum can be compared with demonstrated knowledge measured 
by mathematics and science items in TIMSS tests. 

During the development stages of every TIMSS assessment, a great deal of work 
is dedicated to ensuring comparability of achievement; here, the assessment content 
plays an important role. The test content is agreed by all participating education 
systems working in collaboration and aims to cover topics that are considered relevant 
by participating nations and that are also covered by the prescribed curricula in 
the majority of the participating systems. TIMSS provides additional information 
on the discrepancies between the assessment materials and national curricula, by 
undertaking a test-curriculum matching analysis of content topics (for further details 
and results, see Mullis et al., 2020, Appendix C). We used this data to also assess 
effect of content coverage on regional test scores. 

Learning opportunities are not only provided to students in school but also outside 
formal settings, and especially at home. More books at home and higher education 
levels of parents have been clearly linked with more opportunities for children to 
learn at home (Chiu & Xihua, 2008; Eccles, 2005), but TIMSS provides an even 
more comprehensive scale that can be regarded as a measure of OTL at home, namely 
the TIMSS home resources for learning scale (HRL scale; Yin & Fishbein, 2020, 
p. 16.39). This scale covers the information on number of adult and children books and 
internet access at student home as well as parental education and occupations. Higher 
values on the HRL scale indicate greater access to home resources for learning, which, 
in turn, implies more OTL. 

To investigate the relationship between OTL and achievement in more depth, 
as well as the percentages of students taught the topics, we also considered another 
aspect of OTL which can be linked to the classroom time provided to teaching content. 
In assessing the importance of higher content exposure, we also needed to consider 
how achievement may be affected when classroom teaching and hence content expo- 
sure is limited by the effects of lower home support and more problematic class 


3 Student achievement was measured by a large number of science and mathematics TIMSS assess- 
ment items that together covered all topics from the framework. For TIMSS, the reporting goals 
mean that many more questions are required for the assessment than can be answered by any 
one student in the amount of testing time available. Accordingly, TIMSS uses a matrix sampling 
approach that involves packaging the entire assessment pool of mathematics and science items at 
each grade level into a set of 14 student achievement booklets, with each student completing just 
one booklet and therefore only answering part of the whole set of TIMSS items (Martin et al., 
2017). Item response theory (IRT) and plausible values methodology were used to compare student 
scores on the TIMSS international scale metric, which was set to have a mean of 500 and a standard 
deviation of 100 points in the first cycle of TIMSS (Martin et al., 2020). 
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climate. Among their responses to the TIMSS teacher questionnaire, teachers esti- 
mated how much they felt limited in teaching because of different student attributes. 
This was used to create a TIMSS scale, classroom teaching limited by students not 
ready for instruction (the LSN scale; Fishbein et al., 2021, Supplement 1, pp. 95— 
97). We included this scale in our analyses because additional instructional time is 
expected to be needed for students getting ready for instruction and consequently 
less time is available for teaching content; therefore lower values on this scale can 
be used as a negative measure of OTL. 

To describe and assess the differences in students” opportunities to learn about 
mathematics and science across the Dinaric region, we compared reports from each 
of the participating education systems derived from TIMSS data on the relevant 
factors. We undertook additional analyses to discover the relationships between 
the factors associated with OTL and student achievement, combining data from 
different sources reported on different scales. We calculated average time devoted to 
learning and percentages of the learning time devoted to mathematics and science, 
both according to national prescribed curricula and as reported at the school level by 
principals. We then used teacher reports on the actual content taught in the sampled 
schools and calculated mean percentages of students exposed to specific content for 
each education system. To examine the relationship between students’ opportunity to 
learn and their mathematics and science achievement, we calculated Pearson’s corre- 
lation coefficients between selected variables and student achievement. Finally, we 
used regression models to discover the relations between achievement and content 
exposure, taking two other factors into account: learning outside school and poten- 
tial limitations to exposure to content taught in school. For the first factor, we used 
values on the TIMSS HRL scale to assess students’ external learning opportunities. 
The second variable we used was the TIMSS LSN scale. Student achievement was 
dependent variable in our regression models and we used t-test statistics to determine 
group differences (for a more information about the data sources and methodological 
tools, which account for the complex survey design, please see Sect. 5. 


3 Results 


3.1 Content Exposure 


For all participating Dinaric education systems, we collected available data on 
agreed national curriculum targets or any other official document that prescribed 
the percentage of total instructional time that should be devoted to mathematics and 
science instruction at grade four. As this information was extracted from official 
documents, it can be considered a reasonable indicator of the intended or prescribed 
curriculum at the system level. In some of the Dinaric systems, schools were teaching 
an integrated curriculum (where school subjects are deliberately combined), so hours 
spent solely on mathematics or science instruction could not always be accurately 
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Table : Percentage of Education system Percentage of total teaching time 
instructional time allocated at : 
at grade four prescribed at the 
the system level to DEN 
: : system level for teaching: 
mathematics and science - - 
curricula at grade four Mathematics Science 
Albania 17 9 
Bosnia & Herzegovina 20 10 
Croatia 22 17 
Kosovo 21 8 
Montenegro 19 10 
North Macedonia 22 9 
Serbia 21 8-13 


reported. However, percentages of total instructional time prescribed nationally for 
mathematics and science instruction at grade four indicate that mathematics instruc- 
tion was generally allocated about 20% of total instructional time in the participating 
systems and there was relatively low variation in the amount of time allocated, the 
lowest percentage being in Albania (17%) and the highest in Croatia (2296) (see Table 
2). Science instruction was generally allocated approximately nine percent of total 
instructional time in most of the participating systems, although there variation was 
greater; notably, Croatia allocated 1796 of grade four instructional time to science. 
Mathematics was thus generally allocated more learning time than science, and there 
was also more consistency among the Dinaric systems in the time allocated to this 
topic. 

At the level of curriculum implementation, we compared information collected 
from teachers and principals regarding the instructional time spent in individual 
sampled classes (Fig. 1), as the total instructional time across all subjects per year 
versus the instructional time devoted specifically to mathematics and science in the 
Dinaric education systems. Teachers reported the number of hours of instruction per 
week and principals reported the number of school weeks and days per year. The 
number of hours for mathematics and science per school year show that Croatia and 
Bosnia and Herzegovina provide the highest total number of hours of school instruc- 
tion per year, and Albania and Montenegro provide the lowest number of hours of 
instruction per year. As a comparison, Croatian students received 206 hours more 
instruction per year than their peers in Montenegro. Across the Dinaric region, the 
distribution of time devoted to mathematics and science instruction varied consid- 
erably. Croatia devoted the highest number of hours per year to science instruction 
(82 h), while Kosovo spent the highest number of hours per year on mathematics 
instruction (150 h); Albania devoted the lowest number of hours per year to both 
mathematics and science instruction (113 h and 54 h, respectively). 

We compared information on prescribed hours reported at the system level by 
the national research coordinators with the information provided by principals and 
teachers about the implemented instructional hours and noted there were differences 
across the Dinaric region between the prescribed and implemented instructional 
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Fig. 1 Total instructional time received by students as reported by principals, and instructional 
time devoted to mathematics and science, as reported by teachers. 

Notes In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population. In Kosovo, data for mathematics instruction were available for >70% but <85% of the 
students 


time allocated to mathematics and science. On average, across the Dinaric partici- 
pants, the prescribed instructional time was about 20% of the total instructional time 
for mathematics and about 10% of the total for science (see Table 2). In reality, 
the percentage of implemented instructional time (as calculated from the numbers 
reported by principals and teachers; see Fig. 1) was slightly lower than the prescribed 
instructional time in all participating systems. For mathematics, the time spent on 
instruction ranged from 14% of total instructional time in Croatia and Bosnia and 
Herzegovina to 19% in Kosovo. For science, this time ranged from seven percent in 
North Macedonia to 1196 in Montenegro. 


3.2 Content Coverage for Mathematics and Science Topics 


Our first observation from analyzing the curricular data is that the intended curricula 
in the Dinaric region were defined at the system level and hence consistent for all 
students in each system. According to data collected by the TIMSS 2019 curriculum 
questionnaire (Fig. 2), there were large differences in the intended mathematics topics 
taught across Dinaric systems. Three of them, Albania, Bosnia and Herzegovina, 
and Croatia, indicated that seven out of the 17 TIMSS mathematics topics were not 
included in their grade four curricula. Kosovo reported that four topics were not 
included in their grade four curriculum. There was also considerable variation in the 
selection of topics that were taught. Bosnia and Herzegovina and Croatia flagged four 
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Fig.2 Intended curriculum coverage of TIMSS mathematics topics, as reported by national research 
coordinators. 
Note Numbers in the bars indicate the number of TIMSS topics in each category 


out of seven number content topics were not included in their curricula. Regarding 
measurement and geometry content, Albania and North Macedonia marked four 
out of seven topics as not included in their curricula. It is worth noting that only 
Montenegro included all three data topics in their curriculum; the other Dinaric 
systems did not cover these topics at grade four, or only to minor extent, indicating 
there was generally low coverage of this content domain across the region. In North 
Macedonia, it was noteworthy that a number of topics were intentionally only taught 
to more able students; other Dinaric systems generally expected all topics included 
in the curriculum would be taught to all students (Fig. 2). 

According to the NRC reports, there were also large differences in science 
curricula topics across the Dinaric region (see Fig. 3). Only Kosovo and Montenegro 
reported covering all TIMSS science topics; in North Macedonia there were three 
topics that were only taught to the more able students. Bosnia and Herzegovina and 
Croatia omitted the largest number of TIMSS topics from their curricula (12 and 
10, respectively). Bosnia and Herzegovina, Croatia, and North Macedonia did not 
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Fig.3 Intended curriculum coverage of TIMSS science topics, as reported by national research 
coordinators. 
Note Numbers in the bars indicate the number of TIMSS topics in each category 


cover a large number of the topics belonging to the physical science domain, but 
the other four Dinaric participants reported their curricula covered all twelve topics. 
Montenegro was the only Dinaric participant to cover all TIMSS earth science topics 
in their curriculum (Fig. 3). 

Teachers reported the mathematics content in the TIMSS 2019 assessment that 
they taught their students (Fig. 4). Their reports revealed that there were quite substan- 
tial disparities in the implemented mathematics curricula across the Dinaric region. 
For example, Kosovo, Croatia, and Serbia focused more on measurement and geom- 
etry content, while North Macedonia, Bosnia and Herzegovina, and Albania focused 
more on number content topics. It is noteworthy that teacher reports of implemented 
curricula (Fig. 4) only partially aligned with the intended curricula (Fig. 2). Overall, 
the coverage of the mathematics topics assessed by TIMSS varied across the region, 
with particularly low coverage in Bosnia and Herzegovina, and high coverage in 
North Macedonia (Fig. 4). 
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Fig. 4 Average percentages of topics covered by the TIMSS 2019 mathematics assessment that 
students were taught. 

Notes The 17 TIMSS grade four mathematics topics are described in more detail in Lindquist 
et al., (2017, pp. 15-18). In Kosovo and Serbia, the national defined population covers 90-95% 
of the national target population (for actual percentages and standard errors, see Table S.1 in the 
supplementary materials available for download at https://www.iea.nl/publications/RfEVol13) 


Teachers also reported the science content in the TIMSS 2019 assessment that 
they taught their students (Fig. 5). On average, teachers in Bosnia and Herzegovina, 
Croatia, and Montenegro reported particularly low coverage of the TIMSS science 
topics by grade four, and there was some regional variation in the range of topics 
that were taught. Montenegro, Albania, Bosnia and Herzegovina, and Kosovo had a 
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Fig. 5 Average percentages of topics covered by the TIMSS 2019 science assessment that students 
were taught. 

Notes The 26 TIMSS grade four science topics are described in detail in Centurino and Jones (2017, 
pp. 32-38). In Kosovo and Serbia, the national defined population covers 90-95% of the national 
target population (For actual percentages and standard errors, see Table S.2 in the supplementary 
materials available for download at https://www.iea.nl/publications/RfE Vol 13) 
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greater focus on the life science topics, while Serbia and North Macedonia focused 
more on teaching physical science topics. Serbian teachers seemed to have partic- 
ularly concentrated on teaching physical science content, while Croatia focused 
on teaching earth science content. In Bosnia and Herzegovina, Montenegro, and 
Albania, teacher reports indicated low coverage of earth science topics (39%, 42%, 
and 46%, respectively). 


3.3 Relating OTL with Mathematics and Science 
Achievement 


3.3.1 Relationships Between OTL and Achievement Within Education 
Systems 


The TIMSS data provides an opportunity to compare teachers’ descriptions of content 
taught with the achievement of their students, and thus investigate whether there 
is a relationship between the amount of content taught and student achievement. 
While the design of international large-scale assessments (ILSAs) does not enable 
researchers to draw conclusions about the direct impact of learning factors on achieve- 
ment, it is possible to use bidirectional correlations as indications. However, when we 
analyzed the correlations between teacher reports on content coverage and student 
achievement, we found only spurious statistical evidence of such a relationship across 
Dinaric participants (Tables 3 and 4). 


Table 3 Correlations between teachers’ reports on average content coverage of TIMSS mathematic 
topics and TIMSS mathematics achievement (overall and disaggregated by content domains) 


Education All TIMSS TIMSS number | TIMSS TIMSS data 
system mathematics topics | topics measurement and topics 
geometry topics 


Albania —0.02 (0.04) —0.03 | (0.04) 
Bosnia & 0.03 (0.03) 0.05 | (0.03) 
Herzegovina 

Croatia —0.04 (0.03) 0.00 | (0.04) 
Kosovo* 0.00 (0.04) 0.06 | (0.04) 


Montenegro 0.02 (0.03) 0.04 | (0.03) 0.02 (0.03) 0.00 | (0.03) 


North 0.01 (0.05) 0.02 | (0.06) | —0.02 (0.05) 0.03 | (0.05) 
Macedonia 
Serbia? —0.01 (0.03) —0.05 1(0.03) | —0.01 (0.03) 0.00 | (0.03) 


Notes Statistically significant (p < 0.05) correlation coefficients are shown in bold. Standard errors 
appear in parentheses 
*National defined population covers 90-95% of the national target population 
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Table 4 Correlations between teachers’ reporting on average content coverage of TIMSS science 
topics and TIMSS science achievement (overall and disaggregated by content domains) 


Education system All TIMSS TIMSS life TIMSS TIMSS earth 
science topics | science topics | physical science topics 
science topics 

Albania —0.05 (0.05) 
Bosnia & Herzegovina 0.00 (0.04) 
Croatia 0.02 (0.03) 
Kosovo? —0.03 (0.04) 
Montenegro 0.06 (0.03) 
North Macedonia —0.01 (0.06) 
Serbia? 0.01 (0.05) 


Notes Statistically significant (p « 0.05) correlation coefficients are shown in bold. Standard errors 
appear in parentheses 
*National defined population covers 90-95% of the national target population 


We found that correlation coefficients were very low and, in general, not signifi- 
cant. There was a very weak positive relationship (indicating that higher achievement 
was related to more exposure to the topic) in the mathematics content area of number 
in Bosnia and Herzegovina, and exposure to science content generally, as well as 
earth science content in particular, was positively related to higher student achieve- 
ment in Montenegro. Counterintuitively, in Albania, the more students were exposed 
to life science topics, the lower their scores in this content domain, but this negative 
coefficient was of very low magnitude. Such results are unexpected, but there are 
potential explanations. Teachers may not have been sufficiently confident to confirm 
the basic content as taught, and may have instead chosen the option "not yet taught 
or just introduced". Another explanation may that that the topic as described in the 
TIMSS teacher questionnaire also encompassed topics that were not yet included 
in the curriculum, leading the teacher to again report the topic as not yet taught (a 
specific example might be that the conductivity of heat or electricity within the topic 
of classifying materials based on physical properties was not yet taught, although 
other aspects, such as weight/mass, volume, and state of matter were covered by 
the curriculum). However, additional system-level research is required to establish 
whether such theories are valid. 


3.3.2 Alignment Between OTL and TIMSS Test Materials 
at the System Level 


We referred to the test-curriculum matching analysis result for the Dinaric partici- 
pants (see Mullis et al., 2020, Appendix C) to evaluate whether the content of the 
TIMSS test items was covered in the regional curricula. Achievement scores for 
each education system are recalculated based on the inclusion of only those TIMSS 
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achievement items that were reported as included in the intended curricula up to grade 
four. This information provides additional context regarding a student's OTL, as it 
indicates whether the achievement of a participant would change if only “matched” 
items were considered, and provides a wider perspective on the average national 
achievement, which might be very different if students only responded to curriculum 
matched items. The analysis also reconfirmed the number of items covered by the 
regional curricula. We found large differences in coverage across the Dinaric region; 
as an example, almost all TIMSS science items were covered by the North Mace- 
donian science curriculum, but only half of the science test content was covered by 
the Croatian curriculum (Tables 5 and 6). According our test-curriculum matching 
analyses, across most of the Dinaric participants, in general their mean achievement 
would be the same, even if the TIMSS test had been based on items tailored to their 
specific curricula. There were a few exceptions to this: for example, Montenegro 
and Bosnia and Herzegovina would have performed better if their students had been 
assessed on a selection of items tailored to the Montenegrin curriculum, and students 
in Croatia and Serbia would also have achieved better scores if they had been assessed 
on items covered by the Croatian curriculum. 


3.3.3 The Net Effect of OTL on Student Achievement 


To better understand the varying importance of these different aspects of OTL across 
the Dinaric region, we used the TIMSS topics coverage variables as predictors of 
mathematics and science achievement in regression models, controlling for values 
on the TIMSS HRL and LSN scales (Tables 7 and 8). The LSN scale and TIMSS 
topics coverage variables were reported at the class level by their teachers, but treated 
here as student-level features. The results confirm that there were large differences 
across the region in the relation between content exposure and achievement, even 
after controlling for limitations related to home support and classroom climate, but, 
in general, content exposure was not significantly related to achievement (Tables 7 
and 8). After controlling for the factors attributable to lower home support and limi- 
tations for teaching due to students not ready for instruction, there was no significant 
positive relationship between the percentages of students that were taught the math- 
ematics or science topics and student achievement in any of the education systems 
that we studied. However, in Bosnia and Herzegovina, Montenegro, and North Mace- 
donia, we found that lower scores on the LSN scale were weakly associated with 
higher TIMSS mathematics and science achievement. Across the Dinaric region, 
the strongest significant predictor of higher student achievement in mathematics and 
science was having access to more home resources for learning. This association was 
strongest in Serbia and North Macedonia, but relatively weak in Kosovo, indicating 
that achievement in Kosovo depends less on factors that lie outside formal school 
settings than it does in other parts of the region. 
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Table 5 Test-curriculum matching analysis with the content of the TIMSS 2019 test materials for 
mathematics curricula across the Dinaric region 


TIMSS mathematics 


Education 
system 


Croatia 
Serbia 


Croatia | Serbia 


Number of 
TIMSS test 
items 
covered 
(possible 
TIMSS 
points) 


127 
(136) 


TIMSS less di 


fficult mathematics 


Education 
system 


Albania 


Bosnia & 
Herzegovina 


Kosovo | Montenegro 


North 
Macedonia 


Albania 


Bosnia & 
Herzegovina 


Kosovo 


Montenegro 


North 
Macedonia 


Number of 
TIMSS 
test items 
covered 
(possible 
TIMSS 
points) 


123 (130) 


140 (151) 


177 (190) 


Notes The TIMSS mathematics assessment contained a total of 171 items (students could score a 
maximum of 183 possible assigned points) and the TIMSS less difficult mathematics assessment 
contained a total of 177 items (students could score a maximum of 190 possible assigned points). 
Read down the column under a participant’s name to compare difference in achievement scores 
based on the items identified as covered by that participant. Scores on the diagonal are differences 
in achievement scores, based on the test items identified 
+Mean score would be higher on the item list covered by the compared participant (for example, 
both Montenegro and Bosnia & Herzegovina achieved better scores in mathematics when assessed 
using a selection of test items tailored to the Montenegrin curriculum) 

@ Mean score would not differ on the item list covered by compared participant 
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Table6 Test-curriculum matching analysis with the content of the TIMSS 2019 test materials for 
science curricula across the Dinaric region 


Education Albania | Bosnia & Croatia | Kosovo | Montenegro | North Serbia 
system Herzegovina Macedonia 


Albania e e e e 
Bosnia & e e e e 


Herzegovina 


Croatia 


Kosovo 


Montenegro 


North 
Macedonia 


Serbia e e e e e e e 


Number of 146 100 (104) 79 (81) | 168 124 (128) 168(173) |125 
TIMSS test | (151) (173) (130) 
items 
covered 
(possible 
TIMSS 
points) 


e" eo 


Notes The TIMSS science assessment contained a total of 169 items (students could score a 
maximum of 174 possible assigned points). Read down the column under a participant's name 
to compare difference in achievement scores based on the items identified as covered by that 
participant. Scores on the diagonal are differences in achievement scores, based on the test items 
identified 

+Mean score would be higher on the item list covered by the compared participant (for example, 
both Montenegro and Bosnia & Herzegovina achieved better scores in mathematics when assessed 
using a selection of test items tailored to the Montenegrin curriculum) 

@ Mean score would not differ on the item list covered by compared participant 


4 Discussion 


Our aim was to describe the opportunities to learn provided to grade four students 
across the Dinaric region, and establish whether this could be linked to their 
demonstrated mathematics and science achievement in TIMSS 2019. 

To assess the effect of time exposure, we compared prescribed and implemented 
opportunities to learn. Across the Dinaric region, many of the participants reported 
that the nationally prescribed percentage of teaching allocated to mathematics and 
science was similar and they also reported comparable disparities in the actual imple- 
mentation of the timetable. Like many other participants in TIMSS, official docu- 
ments for the Dinaric region suggested that more time is devoted to teaching mathe- 
matics than science. We also found that participants in the region reported that similar 
percentages of total instruction time were prescribed for mathematics lessons, while 
there was more regional variation in the time spent teaching science. In Croatia, the 
nationally prescribed time for science instruction was at least 5096 higher than in the 
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Table 7 Amount of variance in students’ mathematics achievement explained by the regression 
model, standardized regression coefficients for TIMSS mathematic topics coverage, classroom 
teaching limited by students not ready for instruction, and home resources for learning 


Education Number | Variance | Standardized regression coefficients: 
2 

system of (R^) 2 TIMSS Teaching limited | Home resources 

students explained | mathematics by students not | for learning 

(n) by model topics coverage ready for 

instruction 

Albania 4074 0.16 —0.05 | (0.03) 0.05 (0.04) |0.38 (0.03) 
Bosnia & 5244 0.13 0.05 1(0.03) | 0.06 (0.03) |0.35 (0.02) 
Herzegovina 
Croatia 3631 0.13 —0.02 | (0.03) | 0.01 (0.03) | 0.36 | (0.02) 
Kosovo* 4203 0.09 0.02 | (0.04) | 0.04 (0.03) |0.29 | (0.02) 
Montenegro | 4292 0.13 0.01 (0.02) 0.07 (0.02) | 0.36 (0.01) 
North 2806 0.23 0.04 | (0.03) | 0.13 (0.04) |0.46 | (0.02) 
Macedonia 
Serbia* 4206 0.27 —0.01 | (0.03) | 0.00 (0.03) 0.1 (0.202) 


Notes R2 = the proportion of variance in the outcome variable that is explained by the set of 
predictor variables. Statistically significant (p < 0.05) regression coefficients are shown in bold. 
Standard errors appear in parentheses 

*National defined population covers 90-95% of the national target population 


Table 8 Amount of variance in students’ science achievement explained by the regression model, 
standardized regression coefficients for TIMSS science topics coverage, classroom teaching limited 
by students not ready for instruction, and home resources for learning 


Education Number | Variance | Standardized regression coefficients: 
2 

system of (R) , TIMSS science | Teaching limited | Home resources 

students explained topics coverage | by students not | for learning 

(n) by model ready for 

instruction 

Albania 4074 0.15 0.00 (004 | 0.03 (0.05) | 0.38 (0.03) 
Bosnia & 5267 0.10 0.05 | (0.03) | 0.06 (0.03) | 0.32 (0.02) 
Herzegovina 
Croatia 3631 0.14 0.00 | (0.03) | 0.01 (0.02) | 0.37 (0.03) 
Kosovo* 4270 0.09 —0.02 | (0.04) | 0.05 (0.03) | 0.29 (0.03) 
Montenegro | 4110 0.14 0.06 | (0.03) | 0.06 (0.02) | 0.37 (0.02) 
North 2814 0.26 0.05 | (0.04) | 0.12 (0.04) | 0.49 (0.03) 
Macedonia 
Serbia* 4138 0.26 —0.01 | (0.04) | 0.01 (0.03) | 0.51 (0.03) 


Notes R2 = the proportion of variance in the outcome variable that is explained by the set of 
predictor variables. Statistically significant (p < 0.05) regression coefficients are shown in bold. 
Standard errors appear in parentheses 

*National defined population covers 90-95% of the national target population 
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other participating systems. However, across the Dinaric region, the implemented 
curricula, as reported by teachers and principals, were found to deviate from the 
prescribed teaching time. 

The results from TIMSS 2019 are similar to other sources. According to a Eury- 
dice report (European Commission, 2018) that assessed all grades of primary educa- 
tion across European countries, the bulk of school teaching time was devoted to 
writing and reading, then to mathematics, followed by science. In 2018, in almost all 
European countries at each primary school grade, the number of hours of teaching 
officially recommended for science was significantly less than the number of hours 
recommended for mathematics. According to the report, the percentages of teaching 
hours recommended for mathematics in the Dinaric region at that time ranged 
from 17.9% in Albania to 22.2% in Croatia, and the percentages of teaching hours 
recommended for science ranged from 5.7% in Montenegro to 13% in Bosnia and 
Herzegovina. 

We analyzed NRC reports on nationally prescribed content coverage and 
compared intended curriculum coverage with the list of content topics that were 
assessed by the TIMSS 2019 mathematics and science tests and with teacher reports 
of which TIMSS assessment topics were taught to classes. 

We found that there was lower overall coverage of the TIMSS mathematics content 
than the TIMSS science content. Across the region, in general, we found that more 
than half of the mathematics and science topics were intended to be taught to all 
students. For mathematics, measurement and geometry topics had the best coverage, 
followed by the TIMSS number topics. The three data topics were not covered at all 
by four of the Dinaric participants, while one topic of the three topics was covered 
in another two participants. North Macedonia was the only education system that 
prescribed different content to be taught to more able students. In all other partici- 
pating systems, the same curriculum was delivered to all students. Teacher reports 
on the delivery of topics in in the classroom suggest many topics were covered. On 
average, teachers reported that almost two-thirds of students were taught all TIMSS 
mathematic topics in five of the Dinaric systems, while teachers from Bosnia and 
Herzegovina and Croatia reported fewer students received instruction related to the 
TIMSS content topics. Across the region, the TIMSS content related to the data 
domain was least likely to be covered. 

Regional curricula were better at covering TIMSS science topics than TIMSS 
mathematics topics, but teachers’ reports of classroom delivery of content were more 
inconsistent. In Kosovo and Montenegro, NRCs reported that all of the 26 science 
topics used in the TIMSS assessment were covered in the curricula, and, gener- 
ally, all participants reported good coverage of the science content topics. However, 
unlike mathematics, there were very obvious mismatches between the reports of 
the intended and the implemented curricula. While NRCs reported most science 
topics were covered, teachers from Bosnia and Herzegovina, Croatia and Montenegro 
reported that the percentages of students taught different science content topics in the 
classroom ranged from only 30 percent to 60% and, for the earth science content area, 
the coverage reported by teachers was less than 60% in another three participants. 
When we disaggregated the responses to establish which domains were taught, we 
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found great variation in the attention given to physical science and life science across 
the region. In general, the earth science and physical science topics were less likely 
to be taught than the life science topics. Physical science topics were taught to most 
students in North Macedonia (85% of students) and Serbia (90% of students), while 
the life science topics were taught to most students in Albania (80% of students) and 
Montenegro (75% of students). 

We also analyzed relations between opportunities to learn and learning outcomes. 
While there is ongoing debate about whether increasing instruction time in school 
increases student achievement (Andersen et al., 2016; Jez & Wassmer, 2013; Yesil 
Dağlı, 2019), evidence has suggested that the quality of instruction and the time 
available for learning may have a positive effect on student achievement and even 
compensate for weaknesses in other areas, such as ability or willingness to learn 
(Gettinger, 1985). However, we found that, in the Dinaric region, the TIMSS 2019 
data provided no evidence that spending more hours on mathematics or science led to 
higher achievement. This confirms data from earlier cycles of TIMSS, which showed 
that mean national achievement was not positively associated with average hours of 
instruction (see Martin et al., 2016, exhibit 9.1 and Mullis et al., 2016, exhibit 9.1). 
Our study also showed that, in Dinaric region, there was no unequivocal correlation 
between instructional time and achievement. 

International research shows that other important elements can have a mediating 
effect, such as the quality of instruction or time and opportunities available for 
learning outside school (Jacob & Ryan, 2018; Ozek, 2018). The positive relation- 
ship between increased instruction time and student achievement is more apparent 
when the increase is accompanied by other support measures and directed at disad- 
vantaged students, for instance, those who come from less privileged families or 
home environments (European Commission, 2018). We therefore investigated the 
relationship of OTL with achievement taking into account two mediating variables: 
the accessibility of home resources for learning, as reported by parents in the TIMSS 
home questionnaire, and teacher reports of classroom teaching being limited by 
students who were not ready for instruction, which we used, respectively, as proxy 
measures of opportunities given to students outside school and quality of instruc- 
tion. We found no strong general relationship between content coverage or topics 
taught and achievement. Although some Dinaric systems covered a large proportion 
of topics in their curricula or allocated more time to instruction, these differences 
did not translate into differences in student achievement. The presence of a weak 
relationship between content coverage and achievement showed that, along with 
teachers’ reports of which topics were taught, some students had not yet mastered 
topics reported as taught, and likewise did not know how to solve items that required 
knowledge of content not yet taught in school. The results showed that higher home 
support was an important predictor of higher science and mathematics achievement 
for students in all seven education systems, and an especially strong factor in Serbia 
and North Macedonia. Quality of teaching, as assessed by the teacher reports of 
feeling limited by students not ready for instruction, was found to be significantly 
related to lower achievement in Bosnia and Herzegovina, Montenegro, and North 
Macedonia. This suggests that the science and mathematics capital that students 
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bring with them to school is potentially important. As student achievement seems 
to be strongly linked with opportunities available to them outside school, teachers 
and schools need to be especially aware of the importance of their role in providing 
quality OTL to students who have low levels of home support. 


5 Conclusions 


Our research demonstrated there was some variation in OTL across the Dinaric 
region. We noted discrepancies between intended, implemented, and attained 
curricula in terms of instructional time and content coverage in mathematics and 
science at grade four. Across the Dinaric region, there was no significant common 
relationship between the percentage of students that were taught the topics and mean 
national achievement in TIMSS 2019, but the reports of officially prescribed curric- 
ular contents did not align with teachers’ reports of the content taught in schools. 
School content coverage was not related to achievement. We found achievement 
showed a strong positive association with home learning resources and a weak asso- 
ciation with content exposure in school. Establishing the teaching characteristics 
that could provide the best OTL requires further in-depth research, as learning can 
be facilitated in many ways and may also depend on teacher characteristics. Our 
central message to teachers is that more attention should be given to learning more 
about students’ existing knowledge, whether that was acquired in the classroom or 
outside school. Quality OTL in school is especially important for those students who 
lack the requisite home resources to support their learning. 

This study demonstrates that the concept of OTL is extensive and plays an impor- 
tant role in student achievement. OTL also embraces the opportunities students have 
to learn outside schools. Teachers and policymakers therefore need to consider the 
mathematics and science capital that students are exposed to both inside and outside 
formal school settings, and provide additional school support for those students who 
lack the requisite resources and assistance in the home. Although new national poli- 
cies may be necessary to achieve overall improvement, our analyses of the TIMSS 
2019 data suggest that better teaching and learning of mathematics and science in the 
Dinaric region may be achieved by more rigorous focus on narrowing the disparities 
between the intended and implemented curricula. Understanding the reasons behind 
the observed differences is an important first step. Teachers may wish to carefully 
review the content of the prescribed curricula and compare this with what is presently 
taught in the classroom. 
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Abstract Effective teaching of mathematics and science includes understanding 
the importance of positive attitudes toward learning, and fostering their development 
among students. Many studies have shown that students' motivation to learn is related 
to higher achievement, but when making decisions to improve learning and practice, 
it is important to recognize that cultural influences may also play a role, and estab- 
lishing links between achievement and motivation are thus especially complex. As 
well as measuring student achievement, IEA's Trends in International Mathematics 
and Science Study (TIMSS) thus collects data about these contexts for learning 
through questionnaires completed by students and their parents, teachers, and school 
principals. Data gathered by TIMSS 2019 at grade four indicate that relationships 
between motivation and achievement show many similarities across the Dinaric 
region, and reveal characteristics of the underlying structure of relations between 
attitudes, achievement, and learning support to students in the region. As expected, 
students with more home learning resources tended to show higher mathematics 
and science achievement. Students' confidence in their mathematics or science abil- 
ities tended to be positively correlated with their achievement. Associations between 
mathematics or science achievement and liking learning mathematics and science 
were weaker than the links with reported confidence, but students who reported 
feeling more confident in mathematics or science and those who reported stronger 
feelings of belonging to their school were also more likely to report that they liked 
mathematics and science. There was no strong association between students' home 
learning resources and liking learning subjects; thus indicates that school environ- 
ment plays an important role in supporting motivation for learning. Although similar 
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relations were found across the region, student attitudes in education systems where 
achievement was high tended to be more negative; this is known as the attitudes- 
achievement paradox. The results suggest that further studies of national attitudes 
are needed to better understand local relations between student motivations and 
achievement. 


Keywords Attitudes toward learning : Dinaric region * Grade four education - 
Mathematics : Science : Student achievement - Student motivation * Student 
self-efficacy * Teaching : Trends in International Mathematics and Science Study 
(TIMSS) 


1 Introduction 


Effective teaching of mathematics and science includes understanding the importance 
of developing and fostering positive attitudes toward learning among students. Many 
studies show that students' motivation to learn is associated with higher achievement. 
According to Ryan and Deci (2002), self-determined, or autonomous motivation is 
related to positive academic and emotional outcomes (Um, 2008). Studies in schools 
(Ma & Kishor, 1997; Nicolaidou & Philippou, 2004) indicated that better motivation 
leads to better knowledge of mathematics. However, the connection between attitudes 
and achievement is complex, and measuring motivation across studies is also complex 
(Lee & Stankov, 2018). IEA's Trends in International Mathematics and Science Study 
(TIMSS) has shown, for example, that there were large differences among countries 
in the relationship between mean values on the scales measuring student attitudes 
toward learning and their average achievement over twenty year period from TIMSS 
1995 to 2015 (Mullis et al., 2016b). TIMSS has recognized enjoyment-achievement 
and confidence-achievement paradoxes (Mullis et al., 2016b). In any TIMSS assess- 
ment, there will always be some education systems where mean attitudes toward 
learning and means for the scales measuring student self-confidence are well above 
the TIMSS international average, yet mean student achievement is close to or below 
the TIMSS international average. Conversely, in other education systems, means 
for measures of attitudes toward learning may be relatively low, despite high mean 
student achievement. 

As the number of education systems participating in TIMSS has grown over 
more than two decades, TIMSS has noted that in some high-achieving countries, 
the percentages of students disliking mathematics have increased as achievement 
increased, while in some low-achieving countries, the percentage of students who 
disliked mathematics and achievement both decreased. TIMSS 2015 reported that 
the average correlation coefficient between student confidence and TIMSS mathe- 
matics achievement for grade four students was high (0.67), indicating that high- 
achieving countries also contained larger numbers of students who felt they did not 
usually do well in mathematics (Mullis et al., 2016b). Therefore, to improve educa- 
tion internationally, an understanding of the within-country relationship between 
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students’ attitudes toward learning and their achievement is essential to identify 
which particular aspects of attitudes are most strongly linked to achievement and 
conceive possible explanations for observed relationships. Such in-depth analyses 
can inform strategies designed to help teachers improve specific aspects of student 
motivation. 


2 Background and Research Questions 


We explored the relations between student achievement, student attitudes toward 
mathematics and science, and the support provided by schools and families. When 
making decisions designed to improve learning and practice, identifying which 
factors of motivation are nationally important is an essential first step. Learning from 
other systems with similar cultural backgrounds can help to inform decisions, based 
on effective practices that already exist in the region. Seven participants from the 
Dinaric region took part in TIMSS 2019, namely Albania, Bosnia and Herzegovina, 
Croatia, Kosovo,! Montenegro, North Macedonia, and Serbia. 

Across the Dinaric region, only Serbia and Croatia had participated in both of 
the two previous cycles of TIMSS (2011 and 2015). In TIMSS 2015, the mean 
mathematics achievement of grade four students in Serbia was above the TIMSS 
international average (scale center point); in Croatia, the mean mathematics achieve- 
ment of grade four students was close to the TIMSS international average (the scale 
center point), and hence lower than Serbia (Mullis et al., 2016a). The mean science 
achievement of grade four students was also above the TIMSS 2015 average for both 
entities, while higher in Croatia than in Serbia (Martin et al., 2016). 

TIMSS traditionally measures also trends in student engagement and attitudes 
toward learning of mathematics and science. To accomplish this, data about the 
contexts for learning are collected through questionnaires completed by students 
and their parents, teachers, and school principals (for more information, see TIMSS 
& PIRLS International Study Center, 2018). Responses to items on the student ques- 
tionnaire are used to build the TIMSS scales of liking learning mathematics and 
science (see Yin & Fishbein, 2020, pp. 16.89—16.102). These two scales are each 
modeled from students’ agreements with nine statements on his/her attitudes toward 
mathematics or science, the categories that students can select from being “agree a 
lot," “agree a little,’ “disagree a little,” or “disagree a lot." The resulting continuous 
scales allocate higher scale values for the students who report more positive atti- 
tudes toward learning mathematics or science; these are divided into three sequential 
scale intervals and, consequently, students who mostly or always "agree a lot" were 
categorized as “very much like learning mathematics/science;" those who generally 
“agreed a little" as “like learning mathematics/science,” and all others as “do not like 


! This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 
(United Nations, 1999) and the International Court of Justice (ICJ) Opinion on the Kosovo 
declaration of independence (ICJ, 2010). 


68 B. Japelj Pavešié et al. 


learning mathematics/science,” depending on the score they achieved on the scale 
(see Yin & Fishbein, 2020). 

In TIMSS 2015, about half of the students reported liking mathematics very much 
in Serbia, while this applied only to 29% of students in Croatia, although, for both 
Serbia and Croatia, the mean score on the liking learning mathematics scale increased 
between TIMSS 2011 and TIMSS 2015. This last point is interesting because an 
increase in the mean score for liking learning mathematics was evident in only eight 
of the 49 school systems that participated in both TIMSS 2011 and TIMSS 2015. 
Achievement has increased consistently across all the groups of students by their 
reports of liking learning mathematics (those who do not like, like, or very much 
like) in both school systems. 

TIMSS similarly uses responses to the student questionnaire to define categories 
of students as being “very confident,” “confident,” or “less confident” in mathematics 
or science, based on the ranges of scores on international TIMSS confidence scale 
(see Yin & Fishbein, 2020). Among all the countries and systems participating in 
TIMSS 2015, Serbia had the highest percentage of grade four students who reported 
feeling very confident in mathematics and science (45 and 54%; Mullis et al., 201 6a; 
Martin et al., 2016). In Croatia, only about a third of the students were found to be 
very confident in mathematics, a value close to the TIMSS average, although nearly 
half of students were very confident in science. 

Serbia’s mean score on the liking to learn science scale was at the international 
mean in 2015 and had increased between the TIMSS 2011 cycle and the TIMSS 
2015 cycle, while Croatia’s mean score was below the international mean in 2015 
(Martin et al., 2016). Mean achievement increased more among the students who 
did not like learning or merely liked learning science (in Serbia, by 16 points and 
15 points, respectively, and, in Croatia, by 18 points and 21 points, respectively, on 
a scale with mean of 500 and standard deviation of 100) than among students who 
reported that they liked learning science very much (in Serbia, this increase was three 
points and, in Croatia, 16 points) (Martin et al., 2012). Mean scores on the confident 
in science scale were very high; these were above the TIMSS average both in Serbia 
and Croatia, although Croatia’s mean score decreased between 2011 and 2015. In 
both systems, better achievement was associated with higher confidence in science 
(Martin et al., 2016). Therefore, in these two systems of the Dinaric region, patterns 
of attitudes and achievement appear to differ. Across all the education systems that 
participated in TIMSS 2015, confidence in mathematics was also found to be a 
moderately strong correlate of mathematics achievement for grade four, but liking 
to learn mathematics was only associated with mathematics achievement in some 
systems. Another general attitude toward school, sense of school belonging, was 
weakly related to grade four students’ achievement across all education systems that 
participated in TIMSS 2015 (Lee & Chen, 2019). 

Many researchers have focused on the relationships between student motivation 
and knowledge, especially regarding mathematics. Mata et al. (2012) undertook a 
large meta-analysis study, and found that mathematics achievement was influenced 
by many factors, with student attitudes explaining a significant part of the variance. 
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In another meta-analysis of 113 studies, Ma and Kishor (1997) concluded that rela- 
tions between attitudes toward mathematics and knowledge are generally weak, but 
increase in strength with the age of students. Another study showed that motivation 
could explain almost a third of the variance in mathematics achievement (Lipnevich 
et al., 2011). In Singapore, where the average achievement scores and attitude scale 
scores for students in grade eight are traditionally among the highest of all the educa- 
tion systems that participate in TIMSS, researchers believe that the reasons for those 
positive relations lie within their national curriculum (Fan et al., 2005). Developing 
strong positive attitudes toward learning mathematics is one of the five components of 
Singapore’s mathematics framework and an important goal of teaching mathematics. 

In previous TIMSS studies, comparisons of relations between attitudes and 
achievement in all participating entities revealed that positive associations at the 
student level become negative associations in between-country comparisons of 
mean achievement and attitudes; this is termed the attitudes-achievement paradox 
(first described by Bertling and Kyllonen 2013 in connection with international 
large-scale assessment). The attitudes-achievement paradox has been documented 
across domains and replicated across assessment cohorts (Kennedy & Trong, 2006; 
Kyllonen & Bertling, 2014). Differing explanations have been proposed to explain 
this paradox. In some of the highest achieving participants, this has been related to 
cultural differences reflected in students’ responses to the questions; modesty bias 
or negativity toward the high expectations and academic pressures widely may be 
prevalent in high-achieving Asian countries (Min et al., 2016). Others have suggested 
that a “big-fish-little-pond” effect may be present, where the student’s answers to 
such questions can only be judged relative to the expectations and performance of 
their immediate peers; a student within a group of high-achieving peers may tend 
to report relatively lower confidence and enjoyment in a subject, while the same 
student might report more positive attitudes toward learning if they were placed in 
an environment with lower expectations (Mullis et al., 2016b). Given TIMSS 2019 
provided a unique opportunity to make regional comparisons, we were interested to 
learn whether this paradox was also present across the culturally relatively similar 
Dinaric region; namely, whether higher achievement was associated with less positive 
attitudes toward mathematics and science in comparisons between these education 
systems. 

Seven TIMSS participants from the Dinaric region took part in TIMSS 2019. The 
study data can therefore provide a comprehensive summary of students’ attitudes 
and achievement (and their relationship) across this region. Our main hypotheses 
were that: (a) student achievement was most strongly linked to student motivation 
for learning the subject, and (b) family and school support was positively related to 
attitudes toward learning and student achievement, but (c) there are differences in 
mean achievement, in attitudes, or in relationships between attitudes and achievement 
among participants from the Dinaric region. Our goal was to discover the relation- 
ships between students’ enjoyment of learning about mathematics and science and 
their respective mathematics and science achievement. Identifying the factors related 
to whether or not students like to learn mathematics and science, and which of these 
factors are related to achievement in each participating education system from the 
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Dinaric region provides a critical understanding of differences and commonalities 
among TIMSS participants in the Dinaric region. Better knowledge of the issues can 
support the development of strategies for empowering teachers with an appreciation 
of best methods to enhance positive student motivation within the context of their 
individual education systems. 

Our work was thus guided by four key research questions: 


(1) Do students across the Dinaric region differ in their attitudes toward mathe- 
matics and science? 

(2) Which student attitudes are related to their achievement? 

(3) How are student attitudes related to support provided by their 
parents/guardians and schools? 

(4) Across the Dinaric region, which particular elements of all observed attitudes 
are most strongly linked with student attitudes toward learning mathematics 
and science, and their TIMSS 2019 mathematics and science achievement? 


3 Data and Methods 


We used data collected from students and their parents/guardians from seven partic- 
ipants from the Dinaric region as part of TIMSS 2019. Data includes mathematics 
and science achievement scores, scaled from the students” answers to the TIMSS 
2019 mathematics and science test items, as well as students’ and parents" /guardians' 
responses to questions or statements in the TIMSS context questionnaires (see TIMSS 
& PIRLS International Study Center, 2018) about their learning of mathematics and 
science (Table 1). All these items had four answer categories: “agree a lot,” “agree 
a little," “disagree a little,’ and “disagree a lot.” The answers from all participating 
students in TIMSS were modeled using item response theory (IRT) methods (see 
Sect. 5 and Yin & Fishbein, 2020 for further details) to create different attitudinal 
scales and indexes. 

Socioeconomic status was assessed by collecting information on the learning 
support materials that students could access at home. Students and/or parents were 
asked whether the student had their own room, a study desk, a computer or tablet, a 
mobile telephone for the student to use, family access to internet, and to assess the 
number of books in the home (TIMSS & PIRLS International Study Center, 2018). 
The TIMSS home resources for learning (HRL) scale was derived from students' and 
parents’ answers (Yin and Fishbein 2020, pp. 15.33-15.37). This scale can be used 
to describe learning conditions of students at home in a condensed way. Similar to 
other TIMSS scales, the HRL scale was further simplified by translating it to an index 
variable comprising three levels: “many,” “some,” and “few” resources. We used the 
index variables in the descriptive statistical analyses and the continuous variables of 
scales for the regression analyses. In all our statistical analyses, we followed specific 
requirements for working with international large-scale assessment data, such as 
using weights, sampling errors and procedures for calculations with plausible values 
(see Sect. 5). 
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In some of our descriptive analyses, we combined students assigned to the high- 
and medium-attitude scale categories into one group; students in the lowest attitude 
scale category comprised the comparison group. For example, for many of our anal- 
yses, we simplified things by combined the group of students reporting they liked 
learning mathematics with the group of students that reported they liked learning 
mathematics very much, to form the combined group “students who like learning 
mathematics.” We applied a similar recoding scheme to the indices for students 
like learning science, students confident in mathematics/science, students’ sense of 
belonging, and parents’ perceptions of school performance. 


4 Results 


4.1 Attitudes Across the Dinaric Region 


Among grade four students in the seven TIMSS 2019 participants from the Dinaric 
region, we found that student reports of liking to learn mathematics and science and 
feeling confident in these subjects differed across the region, while students’ sense 
of school belonging and parental perceptions about their child’s school were largely 
similar. In all seven participants from the Dinaric region, almost all students reported 
that they felt that they belonged to their school, and nearly all parents/guardians 
reported positive perceptions of school performance. 

When we separated the data by education system (Fig. 1; see also Table S.3 
in the supplementary materials available for download at www.iea.nl/publications/ 
RfEVol13), we noted similar patterns across attitudes. In Albania, Kosovo, North 
Macedonia, and Montenegro, more students reported liking to learn mathematics 
and science than feeling confident in these subjects. Conversely, in Croatia and 
Serbia, more students reported that they felt confident in mathematics than liking 
to learn mathematics. In Bosnia and Herzegovina, the percentages for these two 
categories did not differ. Overall, in Croatia and Serbia, fewer students reported 
positive attitudes towards learning mathematics than in other Dinaric education 
systems (65% and 74%, respectively, versus >90% in Albania, Kosovo, North Mace- 
donia, and Montenegro). It is encouraging that around 80% of all students across 
the Dinaric region liked learning mathematics and science and also felt confident in 
these subjects. However, there seems to be some discrepancy between confidence and 
achievement, at least in a cross-comparative context, as students' mean performance 
scores in Albania, Croatia, and Serbia were around the TIMSS 2019 international 
average, while students in the other four education systems performed significantly 
below the TIMSS 2019 international average (Mullis et al., 2020). 

When we disaggregated the student distributions within each education system, 
we found that the differences in the relative sizes of the extreme and moderate 
groups were noteworthy. In Croatia and Serbia, the percentages of students who 
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Fig. 1 Percentages of students in each category of six TIMSS 2019 attitude scales: a students 
like learning mathematics, b students like learning science, c students confident in mathematics, 
d students confident in science, e students' sense of school belonging, and f parents' perceptions 
of their child's school. Notes In Kosovo and Serbia, the national defined population covers 90-95% 
of the national target population (See Table 1 for more information about the TIMSS 2019 attitude 
scales (see also Table S.3 in the supplementary materials available for download at www.iea.nl/pub 
lications/RfEVol13) 
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Fig.1 (continued) 


liked learning mathematics or science “very much” and reported feeling “very” confi- 
dent in mathematics or science were smaller than percentages of students who liked 
learning and were confident in mathematics and science. In Croatia, this pattern was 
also repeated for students’ sense of school belonging and parents’ perceptions of 
their child’s school. However, in the other systems across the Dinaric region, this 
situation was reversed, and the extreme categories were chosen by larger percentages 
of respondents than the moderate categories. Students in Croatia and Serbia there- 
fore chose the extreme answers to express their attitudes toward mathematics and 
science notably less frequently than students from other parts of the region; Albanian 
students and their parents/guardians tended to select the most positive attitudes. 
Although these attitude scales may not be directly comparable, the mean values 
reported by these scales provide an overview of prevailing attitudes within these 
education systems (Table 2). In comparing national differences across the region with 
the international mean of 10 points, students’ sense of school belonging was strongest 
in Albania, Montenegro, North Macedonia, and Kosovo, while student confidence 
was strongest in Croatia and parents’ perceptions of their child’s school were most 
positive in Serbia. Across all the participants, attitudes toward science were generally 
less positive than those toward mathematics. Albanian students reported low levels of 
confidence in undertaking both mathematics and science. In Montenegro, Bosnia and 
Herzegovina, and North Macedonia, students reported low confidence in science and 
low levels of liking learning science. In Serbia, students reported low levels for liking 
learning mathematics and science. Albania had the highest mean values for all scales 
and Croatia had the lowest means for all scales. Albania was above the international 
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means for all attitudes, while, in Serbia and Croatia, the means for several attitude 
scales were below the international means. In Croatia, only the student confidence 
scales showed mean values that exceeded the international means, while, in Serbia, 
the attitude scales indicated students tended not to enjoy learning mathematics or 
science, and lacked confidence in science. 


4.2 Student Attitudes Toward Mathematics, Science, 
and Their Achievement 


After examining these different patterns in attitudes, we also anticipated that these 
differences could be related to observed variation in student achievement on the 
TIMSS 2019 mathematics and science tests. We assessed whether and what relation- 
ships existed by calculating Pearson correlation coefficients (r) between student 
attitudes and achievement scores. In general, we observed stronger correlations 
between student achievement and student confidence in mathematics and science 
than between student achievement and students’ liking to learn mathematics and 
science in all participating entities from the Dinaric region (Fig. 2; see also Table SA 
in the supplementary materials available for download at www.iea.nl/publications/ 
RfEVol13). 

According to Cohen's standard for evaluation (see Cohen, 1992; Cohen et al., 
2003), the correlations between mathematics achievement and confidence in math- 
ematics were moderate, ranging from 0.36 to 0.47. There was a difference of only 
around 0.1 between the participant with the highest correlation and the one with 
the lowest correlation, indicating that they were fairly similar. Correlations between 
mathematics achievement and students' liking to learn mathematics were weaker and 
the range of values wider. For students in Bosnia and Herzegovina, Serbia, Croatia, 
and Montenegro, the correlations were generally weak (0.2), while the correla- 
tions with mathematics achievement in North Macedonia and Albania were slightly 
stronger, but still only relatively weakly linked to liking to learn mathematics (0.3). 
Kosovo was the only system where mathematics achievement and liking to learn 
mathematics were moderately linked (a correlation of 0.35). Overall, mathematics 
achievement was consistently more strongly related to confidence in mathematics 
and more weakly related to liking to learn mathematics for all participants from 
the Dinaric region; the trend line also showed that as the strength of the correlation 
of achievement with confidence in mathematics increased, the correlation between 
achievement and liking to learn mathematics tended to decrease (Fig. 2a). 

Similarly, correlations between science achievement and student confidence in 
science were weak for six of the participating entities (ranging from 0.26 in Serbia 
to 0.33 in Kosovo), while North Macedonia exhibited a moderate correlation (0.42). 
The correlations between students' science achievement and liking to learn science 
were also relatively weak, but there was a significant relationship in almost all the 
Dinaric education systems, except for Serbia. North Macedonia showed the highest 
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Fig. 2 Comparing the correlations between student achievement and students like learning scales 
against the correlations between student achievement and the students’ confidence scales for a grade 
four mathematics and b grade four science. Notes In Kosovo and Serbia, the national defined 
population covers 90-95% of the national target population 


correlation coefficient (40.3). Confidence in science was clearly more strongly asso- 
ciated with science achievement than liking to learn science (Fig. 2b). Interest- 
ingly, as opposed to our findings for mathematics, stronger correlations between 
science achievement and confidence in science were associated with stronger corre- 
lations between science achievement and liking to learn science across the region, 
as indicated by the increasing trend between points representing both correlation 
coefficients for each education system (Fig. 2b). 
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In general, our results for the Dinaric region are consistent with previous analyses 
for all systems participating in TIMSS 2015, where average confidence in math- 
ematics was a stronger correlate of average achievement than average scores for 
liking to learn mathematics (Lee & Chen, 2019). The weaker correlations between 
achievement and liking to learn the subject compared to the correlations between 
achievement and confidence similarly reflect results from other countries confirmed 
by previous TIMSS studies (Mullis et al., 2016b). This may be partially because 
confidence scales include the student’s self-evaluation of their knowledge in the 
respective subject. The student’s assessment of their ability is likely to be based on 
prior feedback received about their success in a specific subject, and it is thus not 
unexpected that this would align with their measured achievement. 


4.3 School Environment and Achievement 


As anticipated, the attitudes of students and parents that were not directly linked to 
mathematics or science were less strongly related to student achievement (Table 3). 
Students’ sense of school belonging was only positively correlated with mathematics 
and science achievement in Kosovo; in Serbia the correlation was negative. In Croatia, 
there was a positive correlation between students’ sense of school belonging and 
their science achievement. According to Cohen’s standard (Cohen, 1992; Cohen 
et al., 2003), correlations of <0.2 would have only a small effect. However, Kraft 
(2020) proposed that mean achievement differences of 0.05—0.2 should instead be 


Table 3 Correlations between attitudes toward school and student achievement 


Education Correlation between students’ sense of | Correlation between parents’ 

system school belonging and grade four perceptions of their child’s school and 
student achievement in grade four student achievement in 
Mathematics Science Mathematics Science 

Albania 0.03 (0.02) | 0.04 | (0.03) | —0.05 (0.03) | 20.06 | (0.03) 

Bosnia & 0.02 (0.02) | 0.02 (002) 0.00 (0.02) | 0.01 (0.02) 

Herzegovina 

Croatia 0.03 (0.03) | 0.07 | (0.03) | —0.01 (0.03) | 0.00 | (0.03) 

Kosovo? 0.06 (0.03) | 0.09 | (0.03) | —0.03 (0.02) | —0.02 | (0.02) 

Montenegro 0.01 (0.03) | —0.03 | (0.02) | —0.01 (0.02) | 0.01 (0.02) 

North 0.03 (0.04) | 0.05 | (0.04) | 0.02 (0.02) | 0.02 | (0.02) 

Macedonia 

Serbia? —0.06 (0.03) | —0.07 | (0.03) | 0.07 (0.02) | 0.08 | (0.02) 


Notes Statistically significant (p < 0.05) correlation coefficients are shown in bold. Standard errors 
appear in parentheses 
*National defined population covers 90-95% of the national target population 
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categorized as medium, since cross-country differences tend to be lower than within- 
system differences; in this respect, the correlations here may therefore be regarded 
as having medium effects. 

Parents’ perceptions of school performance, which are based on opinions about 
the effort and success of schools in helping students to achieve their full potential, 
were positively related to both mathematics and science achievement in Serbia and 
negatively related to science achievement in Albania. Although these relations were 
again weak, they indicated that, in Albania, students of parents who have more posi- 
tive perceptions of their child’s school, achieved lower scores in science than students 
of parents who held more negative perceptions of their child’s school. Conversely, 
in Serbia, students with higher mathematics and science achievement in the TIMSS 
tests tended to have parents who reported more positive perceptions of their child’s 
school than the parents of lower achieving students. Students’ sense of belonging and 
parents’ perceptions of school performance were only weakly related to outcomes, 
and the relationship was significant in only a few Dinaric participants. However, 
these findings could arise because there is very little variation in these variables; 
almost all students reported a strong sense of belonging to their school and the vast 
majority of parents felt that their child’s school was doing well (see Fig. 1). 

It is possible that parents may be more satisfied with how the school is working 
if their children present more positive attitudes toward learning and display higher 
levels of confidence in mastering important subjects. However, interestingly, in six 
of the seven Dinaric education systems, parents’ perceptions of school performance 
were negatively related with students’ liking to learn mathematics and science (Table 
4). Moreover, in five of the Dinaric participants, parents’ perceptions of school perfor- 


Table 4 Correlations between parents’ perceptions of school performance and the student attitude 
scales 


Education Correlation between parents’ perceptions of their child’s school and the TIMSS 
system attitude scale 
Students like Students confident | Students like Students 
learning in mathematics learning science | confident in 
mathematics science 
Albania —0.07 (0.03) | —0.08 (0.02) | —0.07 | (0.03) | — 0.06 (0.02) 
Bosnia & —0.09 (0.02) | —0.03 (0.02) | —0.06 | (0.02) | —0.03 (0.02) 
Herzegovina 
Croatia —0.07 (0.02) | —0.07 (0.02) | —0.05 | (0.02) | —0.05 (0.02) 
Kosovo? —0.05 (0.02) | —0.04 (0.02) | —0.08 | (0.02) | —0.06 (0.02) 
Montenegro | —0.11 (0.01) | —0.08 (0.02) | —0.09 (0402) | — 0.08 (0.02) 
North —0.03 (0.02) | —0.03 (0.02) | —0.03 | (0.02) | —0.01 (0.02) 
Macedonia 
Serbia* —0.06 (0.02) | —0.04 (0.02) | —0.04 | (0.02) | —0.02 (0.02) 


Notes Statistically significant (p « 0.05) correlation coefficients are shown in bold. Standard errors 
appear in parentheses 
? National defined population covers 90-95% of the national target population 
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mance were negatively related with student confidence in mathematics and, in four 
participants, negatively related to student confidence in science. 

These results are similar to a previous study of six parental involvement vari- 
ables from TIMSS 2015 in 18 European countries (Koršñáková & Stefanik, 2019). 
Parental perceptions of school performance were the weakest predictors of student 
mathematics achievement, and generally non-significant, while home resources for 
learning was the strongest and most significant predictor of mathematics achievement 
in all the 18 selected European TIMSS participants. 

In order to obtain a more detailed picture of parental perspectives, we disaggre- 
gated the percentages of students with parents who agreed "a lot" with the various 
statements about their child's school performance that were included on the TIMSS 
scale, parents perceptions of their child school (Fig. 3). In most TIMSS participants 
from the Dinaric region, parents generally agreed a lot with the statement that the 
school does a good job in helping the child to become better in reading, mathe- 
matics, and science, and they also seem to be mostly satisfied about being informed 
on progress, being included in the child's education, and that schools care about the 
child's progress. However, fewer parents agreed that the school provided a safe envi- 
ronment and, alarmingly, far fewer parents agreed that their child's school promoted 
high academic standards. The finding that parents in Croatia were less likely to agree 
alot with the statements than parents in other parts of the region aligns with the gener- 
ally larger percentages of students who were assigned to intermediate responses on 
the parents' perceptions of their child's school scale (Fig. 1) and with less positive 
attitudes in general in Croatia (Table 2). 

Looking into these relations in more depth revealed other important facts about 
student attitudes. First, among all TIMSS participants in the Dinaric region, the 
relations between student's sense of school belonging with liking to learn or feeling 
confident in both mathematics and science were all significant and positive (Table 4). 
In other words, students who felt more attached to their school also tended to report 
greater enjoyment in learning and greater confidence in their capabilities, although 
we did note relatively large differences in the strength of these relations among the 
participating systems. In Serbia, Bosnia and Herzegovina, Montenegro, and Croatia, 
relations between students' sense of school belonging and liking to learn science, as 
well as liking to learn mathematics, were moderate to strong (correlation coefficient 
20.3), while in all seven participants, the relations between students” sense of school 
belonging and students' confidence in both mathematics and science were weak, but 
nevertheless significant (Table 5). 

Student attitudes appear to be more strongly connected to their achievement than 
their parents' attitudes toward their school. When defining priorities for improve- 
ments in learning practices at school and system level, it is always important to 
remember that the perceptions of students may differ from those of their parents, 
although the results here are consistent with the proposition that the quality of the 
school environment matters. 
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Fig.3 Percentages of students whose parents agree “a lot” that their child’s school: a does a good 
job in helping him/her become better in reading; b does a good job informing me of his/her progress; 
€ does a good job in helping him/her become better in mathematics; d does a good job including 
me in my child's education; e does a good job in helping him/her become better in science; f cares 
about my child's progress in school; g provides a safe environment; and h promotes high academic 
standards. Note In Kosovo and Serbia, the national defined population covers 90-95% of the national 
target population 
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(g) My child’s school provides a safe environment (h) My child’s school promotes high 
academic standards 
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Fig.3 (continued) 


Table5 Correlations between students’ sense of school belonging and other student attitude scales 


Education Correlation between students’ sense of school belonging and the attitude scale 

system Students like Students like Students confident | Students 
learning learning science | in mathematics confident in 
mathematics science 

Albania 0.23 (0.02) | 0.22 (0.02) | 0.15 (0.02) | 0.13 (0.02) 

Bosnia & 0.45 (0.02) | 0.42 (0.01) | 0.22 (0.02) | 0.29 (0.02) 

Herzegovina 

Croatia 0.41 (0.03) | 0.39 (0.03) | 0.27 (0.04) | 0.27 (0.03) 

Kosovo* 0.17 (0.02) | 0.22 (0.02) | 0.11 (0.02) | 0.18 (0.02) 

Montenegro | 0.43 (0.01) | 0.34 (0.02) | 0.22 (0.02) | 0.22 (0.02) 

North 0.26 (0.03) | 0.27 (0.03) | 0.15 (0.02) | 0.17 (0.02) 

Macedonia 

Serbia* 0.48 (0.02) | 0.42 (0.02) | 0.26 (0.02) | 0.28 (0.02) 


Notes Statistically significant (p < 0.05) correlation coefficients are shown in bold. Standard errors 
appear in parentheses 
*National defined population covers 90-95% of the national target population 


4.4 Relations Between Achievement and Attitudes 


We derived between-system comparisons of attitudes and achievement by comparing 
mean scores on the TIMSS attitude scales with the mean achievement score for each 
education system. Note that the means of different attitude scales within education 
systems can also be influenced by differing cultural interpretation of the questions, 
and thus comparisons are tentative. However, in general, we observed that lower mean 
attitude scale scores were associated with higher mean achievement scores in both 
mathematics and science, which suggests that the attitude-achievement paradox was 
present across the Dinaric region (Fig. 4). Albania was an exception to this pattern, 
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Fig.4 Between-system comparisons of TIMSS 2019 achievement scores and attitudes for a grade 
four mathematics, and b grade four science. Notes In Kosovo and Serbia, the national defined 
population covers 90-95% of the national target population 


recording both relatively high attitude scores and high student achievement; it is 
perhaps noteworthy that Albania was the only Dinaric participant that was not part 
of the former Yugoslavia. 

To identify which factors may be linked to better learning environments and 
higher achievement in each education system, a more comprehensive framework 
is required. We therefore used a set of factors that describe students’ background 
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and learning environments and applied four regression models to further analyze 
student motivations to learn mathematics and science and student achievement in 
those subjects. In these regression models, factors related to the learning environment 
were students’ sense of school belonging and their parents’ perceptions of school 
performance. Factors related to student background were student gender and family 
socioeconomic status, as well as students’ confidence in mathematics and science. 

We assessed the socioeconomic status of students’ families across TIMSS Dinaric 
participants using the TIMSS 2019 HRL scale. In six of the seven Dinaric education 
systems, large majorities (>80% of students) belonged to the intermediate category 
of students that possessed some resources at home; Albania was the exception, where 
only 65% of students fell into this intermediate category. The percentages of students 
having many resources at home ranged from four percent (Kosovo) to 13% (Serbia). 
The percentage of students having only few resources at home were relatively low 
for six of the participating systems, ranging from three percent (Croatia) to 15% 
(North Macedonia); Albania was again the exception, with a more substantial portion 
of students (30%) having few home resources for learning. Internet access is an 
important element of the HLR scale, and it is noteworthy that 36% of Albanian 
students did not have internet access at home; in other parts of the Dinaric region, 
the percentage of students without internet at home was much smaller. 

We then analyzed students’ liking to learn mathematics in terms of the five factors 
that we identified as related to the learning environment and student background. 
These factors explained a substantial percentage of the variance in liking to learn 
mathematics in our model (the variance was >40% in four of the seven education 
systems), from 21% in Kosovo to 53% in Serbia (Table 6). We also estimated the 
standardized regression coefficients for each factor (Table 6). Regression coefficients 
can provide greater understanding of the observed variance, indicating: (a) whether 
the factor makes a significant contribution toward explaining the variance in students’ 
liking to learn mathematics, and (b) the relative strength of the relation between the 
individual factor and students’ liking to learn mathematics, when all other factors 
are kept constant. We made boys the reference group for the coefficient on gender, 
thus negative values in our model indicated that girls were less likely to like to learn 
mathematics than boys, and vice versa. 

For all participants from the Dinaric region, we found that confidence in mathe- 
matics was the factor that was most strongly related to liking to learn mathematics, 
followed by students’ sense of school belonging (Table 6). The other three factors 
were significantly related to liking to learn mathematics in only some education 
systems, but regression coefficients were very small and thus, generally, the rele- 
vance of these three factors in the model was almost negligible. Student confidence 
in mathematics and their sense of school belonging seem to be much more strongly 
related to positive attitudes toward learning mathematics, and thus seem to be more 
strongly associated with successful learning. 

We found similar results when we used regression modeling to analyze the rela- 
tions of these five factors with students’ liking to learn science (Table 7). Again, these 
factors explained a substantial percentage of the variance in liking to learn science 
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in our model (the variance was again >40% in four of the seven education systems), 
and ranged from 29% (Kosovo) to 47% (Serbia) across the Dinaric region. 

The model results again indicated that student confidence in science was most 
strongly associated with students” liking to learn science. Across the Dinaric region, 
students” sense of school belonging was again also strongly related to liking to learn 
science, but the standardized regression coefficients indicated that this factor made 
a smaller contribution toward explaining the variance than confidence. The other 
factors were significantly related to liking learning science in only a few partici- 
pants and even when significant, the analyses showed relations were weak. Parents’ 
perceptions of their child’s school were related to liking to learn science among 
students in Kosovo, but the relation was negative, meaning that the more dissat- 
isfied parents are with the performance of the child’s school, the more students’ 
like learning science (see Sect. 4.4). Family possession of learning resources was 
related to students’ liking to learn science in five of the seven Dinaric participants. In 
Kosovo, having more home learning resources was positively related to liking to learn 
science. In Bosnia and Herzegovina, Croatia, Montenegro, and Serbia, having more 
home learning resources was negatively related to liking to learn science, a finding 
that is counter-intuitive and warrants further research. We note that our regression 
analysis model controlled for factors such as confidence and sense of belonging, 
but the simple direct correlation coefficient between home learning resources and 
liking to learn science was negative in Bosnia and Herzegovina (—0.1 + 0.02), and 
not significant in Croatia or Montenegro; it only became a positive, but small value 
(0.1 + 0.02) in Serbia. One explanation may be that families with high levels of 
home learning support may put more academic pressure on students, which in turn 
decreases their motivation toward learning, but it may also be that families with low 
levels of home learning resources place a high value on education and more strongly 
emphasize the importance of learning to their children. 

We also modeled the relations among the five factors and liking to learn with 
TIMSS mathematics and science achievement (Table 8). In four school systems, the 
combination of the six factors explained considerable percentages of the variance 
in mathematics achievement (22—41 % across the Dinaric region; see Table 8). In all 
the education systems, students’ confidence in mathematics was positively related 
to their mathematics achievement when controlling for all other factors, but varied 
from being strongly related to mathematics achievement in Croatia, Serbia, and 
Bosnia and Herzegovina, to more weakly related to mathematics achievement in 
Kosovo and Albania. The second factor that was strongly related to mathematics 
achievement was the socioeconomic status of students’ families as assessed by their 
home resources for learning. All the relations were positive, indicating that more 
home resources for learning and greater confidence can both be positively associated 
with higher achievement in TIMSS. Liking to learn mathematics was positively 
related to mathematics achievement in three participants, most strongly in Kosovo. 
However, in Bosnia and Herzegovina, students with lower mathematics achievement 
scores reported that they liked learning mathematics more than their high-achieving 
peers. 
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In general, we found differing results across the seven Dinaric participants, 
leading to a range of different conclusions and interpretations about how students’ 
learning environments and attitudes toward learning may affect student outcomes. 
For example, in Bosnia and Herzegovina, Croatia, and Kosovo, our modeling indi- 
cated that boys achieved higher TIMSS mathematics scores keeping all other factors 
constant, but, in the other four school systems, there were no gender differences in 
average mathematics achievement. 

In Albania, mathematics achievement tended to be positively related to home 
learning resources, then to students’ confidence, followed by liking to learn mathe- 
matics, but the analysis indicated that mathematics achievement was related nega- 
tively, albeit weakly, to students’ sense of school belonging. In Bosnia and Herze- 
govina, mathematics achievement was most strongly positively related to students’ 
confidence in mathematics, and then to home learning resources, while the relations 
with liking to learn mathematics and gender were both weakly negative. In Croatia, 
Montenegro, and Serbia, student achievement was most strongly positively related 
to students’ confidence and slightly less strongly related to more home learning 
resources, but there was also a weakly negative relationship with students’ sense of 
school belonging. In Croatia, gender was also related to mathematics achievement, 
with female students tending to score lower in TIMSS than male students. In Kosovo, 
three predictors showed similar strong relations with mathematics achievement: 
liking mathematics, feeling confident, and having more home learning resources. As 
in Croatia, female students tended to score lower on the mathematics test. In North 
Macedonia, home learning resources was the strongest positively related predictor of 
mathematics achievement, followed by a less positive relation with students’ confi- 
dence; while there was also a positive relation to liking to learn mathematics, this 
was very weak. 

We found that science achievement was generally less strongly related to the 
factors included in our analyses. In all participating school systems, the strongest 
predictor of higher science achievement was having more home learning resources 
(Table 9). This predictor was strongest in Serbia, while Kosovo and Bosnia and 
Herzegovina showed home resources for learning was a much weaker predictor of 
achievement. Students’ confidence in science was also positively related to higher 
achievement in all TIMSS participants from the Dinaric region. Girls achieved higher 
scores in North Macedonia, but modeling showed there were no gender differences 
in achievement in other parts of the region. In Serbia, liking to learning science 
and sense of school belonging had negative relations with achievement, indicating 
students who scored highly on these scales tended to have lower science achievement 
scores. In Montenegro, the relation between sense of school belonging and science 
achievement was also weakly negative. 
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5 Discussion 


Our starting hypothesis that student achievement is most strongly linked to student 
motivation for learning the subject could be only partially supported, as we found 
that achievement was most strongly linked to student confidence in a subject. Our 
hypothesis that more family and school support for learning would be related to 
more positive student attitudes and better achievement could not be supported since, 
although sense of school belonging can be counted as school support and tended 
to be linked to positive attitudes toward mathematics and science, home learning 
resources were mostly negatively associated with students’ liking of mathematics 
and science. Increased family and school support for learning was also generally 
unrelated to higher student achievement; while home learning resources were posi- 
tively associated with achievement, there was no consistent strong positive relation 
with sense of school belonging. 

In comparison to other systems in the Dinaric region, Albanian students had 
very strong positive attitudes toward mathematics and science, but, conversely, they 
reported the lowest confidence in mathematics and science, and their science achieve- 
ment was negatively related to parents’ perceptions of their school. When assuming 
all other factors were equal for all students, higher mathematics achievement was 
predicted for students who more often reported liking to learn mathematics, who 
were more confident in mathematics, and who had more home learning resources, 
but, curiously, achievement was also related to a lower sense of school belonging. 
Higher science achievement was predicted for students who were more confident in 
science and had more home learning resources. 

In Bosnia and Herzegovina, the percentages of students who liked mathematics 
and science were similar to the percentages of students who felt confident about 
the subjects, but all these percentages were lower than the percentages of students 
who had a positive sense of school belonging and had parents who reported positive 
perceptions of their child’s school. Higher mathematics and science achievement 
were negatively related to liking to learn mathematics or science but positively related 
to student confidence and access to home learning resources. It was interesting that 
students with lower mathematics achievement scores reported that they liked learning 
mathematics more than their high-achieving peers. One explanation may be that less 
privileged students get more attention from their teachers, and that this investment 
by their teachers consequently increases their enjoyment in learning mathematics. 

In Croatia, the percentage of students who liked to learn mathematics was the 
lowest among all participating Dinaric school systems, and they tended to report the 
lowest percentages of positive attitudes in all our analyses. While science achieve- 
ment was positively related to students’ sense of school belonging, mathematics 
achievement was negatively associated with students’ sense of school belonging. 
Also noteworthy were the generally low levels of satisfaction reported by Croatian 
parents with their child’s school when compared to the other TIMSS participants in 
the region; this potentially merits further research at a national level. 
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In Kosovo, of the percentages of students expressing confidence in learning were 
low (and lower for science than for mathematics) when compared to the percentages 
of students sharing other attitudes, which were generally highly positive. Mathe- 
matics and science achievement were positively related to students’ sense of school 
belonging. Students who were more likely to report liking to learn mathematics 
or science, who were more confident, and who had more home learning resources 
tended to score more highly in TIMSS. 

In Montenegro, students were less likely to feel confident than to like learning 
about mathematics, and especially science, which they like to learn very much. 
Almost all students shared a positive sense of school belonging and almost all parents 
held positive perceptions of their child’s school. 

In North Macedonia, the percentages of students who liked to learn mathematics 
and science were higher than the percentages of students who felt confident in these 
subjects, and close to the high percentages of students with a positive sense of school 
belonging and had parents who held positive perceptions of their child’s school. 

In comparison, Serbia demonstrated large differences in attitudes, with relatively 
low percentages of students liking to learn or being confident in mathematics or 
science and high percentages of students with a positive sense of school belonging 
and with parents who held positive perceptions of their child’s school performance. 
Mathematics and science achievement were negatively related to students’ sense of 
school belonging, but positively related with parents’ perceptions of their child’s 
school. 

Overall, our results indicate that, in Croatia, North Macedonia, Montenegro, and 
Serbia, students who were more confident in mathematics or science and who had 
more home learning resources tended to achieve higher mathematics and science 
TIMSS scores. Higher achievement also showed a negative association with students’ 
sense of school belonging in Montenegro and Serbia. 

In addition, assuming that all other factors are equal for all students, boys tended 
to score more highly in mathematics than girls in Bosnia and Herzegovina, Croatia, 
and Kosovo, while girls tended to score more highly in science than boys in North 
Macedonia. 

In general, we found that achievement was not directly linked to students’ sense 
of school belonging, while the associations with parental perceptions of their child’s 
school were unexpected and not easy to explain with our multivariate analyses. 
However, we can suggest some explanations for the weak relations that we observed. 
In line with world trends, school systems across the region have focused more on 
issues of equality and equity in education. Therefore, more emphasis has been given 
to providing better learning conditions for underprivileged students, which could, 
in turn, create more positive perceptions of schools among parents and a stronger 
sense of school belonging among underprivileged and lower achieving students. Our 
analysis of parental perceptions suggests that policymakers may wish to focus on how 
schools can better involve parents in their child’s education in some parts of the region. 
In addition, the TIMSS scale for sense of school belonging also contains items related 
to students’ feelings of safety; responses to such items may more closely reflect 
concepts of not being bullied than environments supporting achievement. Further 
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study needs to examine the reasons for the relatively weak observed correlations 
between students feeling that they are not bullied and their sense of school belonging 
(this correlation was 0.20 in Croatia, Kosovo, and North Macedonia, 0.22 in Serbia 
and Bosnia and Herzegovina, 0.23 in Montenegro, and 0.25 in Albania). 


6 Conclusions 


We found that the relations among different attitudes across the Dinaric region were 
complex, and that there was some evidence for the attitude-achievement paradox, 
identified previously in other parts of the world (Min et al., 2016). For all Dinaric 
TIMSS participants, the relations between students” sense of school belonging and 
both liking to learn and feeling confident in mathematics and science were significant 
and positive. The percentages of students that had a positive sense of school belonging 
and students with parents that had positive opinions about their child’s school were 
high across the Dinaric region (>90% for both). This implies strongly that schools 
across the region are respected and valued institutions, with great power to support 
change and help stabilize society. 

However, the percentages of students with positive attitudes toward learning both 
subjects and feeling confident in these subjects differed somewhat across the Dinaric 
region. We observed regional variations in the levels of positive attitudes of students 
had toward learning and their school climate. We did not anticipate that more positive 
attitudes of students’ towards learning science or mathematics would generally be 
related to more positive parental perceptions of school performance. In most of 
the TIMSS participants from the Dinaric region, we found more positive students’ 
attitudes among students with less home resources for learning than among students 
with more home resources for learning (which is a measure of the family support). 
These findings further support the important roles that school climate and teachers 
play in motivating students for learning as part of the learning process in schools. 

The relationships between attitudes and achievement are complex. We found that 
while students may report positive attitudes toward learning both mathematics and 
science, enjoyment of learning is not the strongest predictor of achievement in the 
Dinaric region. Student achievement was much more strongly correlated with confi- 
dence in mathematics or science than with liking to learn these subjects. As expected, 
higher achievement was also associated with higher levels of home resources for 
learning in all school systems. However, the associations between achievement and 
students’ sense of school belonging and/or parents’ perceptions of their child’s school 
differed across the region and were only significant in a few cases. These findings 
are in accordance with previous results from TIMSS 2015, which showed that confi- 
dence is a strong correlate of achievement while sense of school belonging is not. As 
a school’s emphasis on academic success is highly related to student achievement 
(Mullis et al., 2016a), a focus on developing a supportive school climate should be at 
the heart of principals” and policymakers' efforts to improve teaching and learning 
for students. 
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What to do with these findings? Our results suggest that, to foster better achieve- 
ment, the Dinaric school systems should recognize that students’ confidence in math- 
ematics and science seems to be a more important factor than their enjoyment in 
learning the subjects. Although we assume that many teachers and schools already 
encourage positive attitudes toward learning mathematics and science among their 
students, and that fostering an enjoyment of mathematics and science is addressed in 
student textbooks and promoted in new teaching approaches and by the use of tech- 
nology, this analysis of the TIMSS 2019 data suggests that more can be done. Teachers 
also need to identify strategies and teaching approaches that develop students’ confi- 
dence in their ability to learn and the application of their knowledge and skills. 
They also need to recognize that, while the factors linked to positive attitudes toward 
learning (such as gender or parental attitudes) may differ for each school system, 
a sense of school belonging seems to play a consistently important role in student 
enjoyment of mathematics and science. While the availability of home resources for 
learning is a key predictor of student achievement, it is a poor predictor of student 
attitudes like enjoyment of learning or self-efficacy. 

Our analyses provide some important general messages for policymakers across 
the Dinaric region: strong factors exist in schools and classes that have an influence on 
students’ interest, enjoyment, and knowledge of mathematics and science. Schools 
and teachers can benefit from a better understanding of the varying contributions 
of competing factors acting within their education system, and there are clearly 
advantages to sharing inspiring examples and successes from neighboring school 
systems when planning stimulating school environments. Education systems may 
need to work on changing common misconceptions about the associations between 
attitudes toward education and achievement if they are to turn their efforts into 
effective improvements for learning. The varying associations that we observed by 
analyzing the TIMSS 2019 data for the school systems of the Dinaric region, and 
comparison of their relative strengths or weaknesses can help to determine specific 
policy measures to help address problems. 
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Abstract Studies have indicated that early literacy (EL) and early numeracy (EN) 
competencies are strong predictors of later mathematical performance in school. 
Data from IEA’s Trends in International Mathematics and Science Study (TIMSS) 
2019, together with comprehensive exploration of regional similarities and differ- 
ences between education systems, confirm that students’ preschool EL and EN 
competencies are important predictors of mathematics achievement among grade 
four students from the Dinaric region. This applies for all content domains specified 
in the TIMSS 2019 mathematics framework: numbers, measurement and geometry, 
and data. Although TIMSS 2019 parental reports for the different EL and EN tasks 
varied considerably across the region, children in the Dinaric region who could recog- 
nize letters, write numbers, or count independently before starting school tended to 
achieve higher scores on the mathematics tasks in TIMSS 2019. This confirms that 
EL and EN skills have a strong relationship with later school outcomes in math- 
ematics. Recognition of these findings could provide the basis for changes in the 
preschool curriculum and further development of programs for parents/guardians on 
numeracy development. 
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1 Introduction 


Early literacy and numeracy competencies relate to the set of knowledge and skills 
developed during the preschool period. Early literacy (EL) includes the knowledge 
and abilities linked to language (vocabulary, background knowledge, semantics, and 
communication skills), print awareness (alphabet, and concepts about print), and 
phonological awareness (rhyme, alliteration, segmentation, and blending) (Rohde, 
2015). Early numeracy (EN) is a term that encompasses several skills, such as verbal 
counting, knowing number symbols, recognizing quantities, discerning number 
patterns, comparing numerical magnitudes, and manipulating quantities (1.e., adding 
and subtracting objects from a set) (Raghubar & Barnes, 2017). EL and EN compe- 
tencies, as a part of a school readiness construct, have been demonstrated to be strong 
predictors of mathematical achievement in school (Duncan et al., 2007; Melhuish 
et al., 2008; Nguyena et al., 2016). 

Itis important to emphasize that both EL and EN are set in the context of cultural, 
demographic, and community characteristics. Thus, they can be viewed as an inter- 
active process of skills and context rather than a linear series of individual compo- 
nents (Rohde, 2015). Numerous studies have confirmed this theoretical stance, indi- 
cating that children's exposure to literacy and numeracy experiences in the preschool 
period may be positively correlated with their early literacy and numeracy compe- 
tencies (Gustafsson et al., 2013; LeFevre et al., 2009; Skwarchuk et al., 2014). It is 
suggested that several factors account for this effect: socioeconomic status (SES) of 
families, number of years spent in kindergarten, home resources related to language 
and mathematical competencies, parental preschool literacy and numeracy practices, 
and parental attitudes toward mathematical development and schooling in general 
(Zippert & Rittle-Johnson, 2020). Cultural differences can play an important role in 
the quality of home numeracy experiences (Aunio et al., 2004; Lefevre et al., 2002). 
They are usually associated with sociological differences, such as perceived value 
of education and knowledge in general, educational policies, and parents/guardians’ 
perceptions related to whether children should be learning through school-like activ- 
ities in early childhood, as well as linguistic differences in the way numeral systems 
are represented (Cankaya & LeFevre, 2016). In addition, parental attitudes and beliefs 
about their own, as well as their children's mathematical competencies, can influence 
the nature of the early learning experiences they provide (Hart et al., 2016; Zippert 
& Ramani, 2017). 

In the context of large-scale assessment studies like IEA's Trends in International 
Mathematics and Science Study (TIMSS) and Progress in International Reading 
Literacy Study (PIRLS), recent cycles have indicated that early learning activities can 
help to lay the foundation for positive schooling outcomes in the future (Meinck et al., 
2018). Results from TIMSS 2011 for European Union (EU) countries showed that EL 
and EN competencies were related to subsequent success in mathematics (Soto-Calvo 
& Sánchez-Barrioluengo, 2016). Nevertheless, the strength of the prediction varied 
across the countries, as well as the size of the effect of early numeracy compared to 
early literacy competencies. 
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2 Purpose of the Study and Research Questions 


Seven participants from the Dinaric region took part in TIMSS 2019, namely Albania, 
Bosnia and Herzegovina, Croatia, Kosovo, ! Montenegro, North Macedonia, and 
Serbia. Given the importance of social and cultural diversities, our aim was to extend 
research into the value of developing early competencies using TIMSS 2019 data 
for participants from the Dinaric region. A previous study into early numeracy expe- 
riences in Serbia from TIMSS 2015 showed that the variables related to providing 
a supportive home environment for learning (home resources for learning, early 
literacy and numeracy activities, preprimary education, early literacy and numeracy 
tasks; (see Mullis et al. 2016) explained more than a quarter of the variance in student 
achievement at grade four (Radišić & Seva, 2017). They found that availability of 
home resources for learning proved to be the strongest predictor of achievement, 
followed by a variable based on parents’/guardians’ assessment of their child’s math- 
ematical competencies before starting school, and kindergarten attendance. We based 
our research design on that used for a study of EU countries undertaken by Soto- 
Calvo and Sánchez-Barrioluengo (2016). Using this design enabled us to follow the 
trends in the relation between EN/EL competencies and mathematics achievement 
in Dinaric region in a comparable manner. 
Our work was guided by three key research questions: 


(1) Do students from Dinaric region exhibit regional differences in early numeracy 
and literacy competencies? 

(2) To what extent is student performance in mathematics at grade four related to 
early literacy and numeracy competencies, and, in particular, related to the 
tasks from number content domain? 

(3) Are there any gender differences in levels of early literacy and numeracy 
competencies, and are those differences reflected in student mathematics 
achievement at grade four? 


3 Variables 


The predictor variables used in this study are the composite variables early literacy 
tasks (ELT) and early numeracy tasks (ENT). These variables represent parental esti- 
mation of their children's competencies before entering the first grade of primary 
school regarding their early literacy and early mathematical competencies, collected 
retrospectively when the students were in grade four through the TIMSS 2019 
Early Learning Survey (also referred to as the home questionnaire; TIMSS & 
PIRLS International Study Center, 2018). We also used parental reports of single 
literacy and numeracy tasks to explore specific characteristics of students' preschool 
competencies in more detail (Table 1). 


! AU references to Kosovo in this document should be understood to be in the context of United 
Security Council resolution 1244 (1999). 
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Table 1 List of variables used in analyses 


Z. Džumhur et al. 


Variable 


Early literacy tasks 
scale? 


Description 


Parents’ responses on 
how well their child 
could do the following 
when he/she began the 
first grade of 
primary/elementary 
school: 

* Recognize most of 
the letters of the 
alphabet 

* Read some words 

* Read sentences 

* Read a story 

* Write letters of the 
alphabet 

* Write his/her name 

Response options: very 

well, moderately well, 

not very well, not at all 


Values/Response 
options 


The higher the score 
the higher the 
student's competency 
in performing early 
literacy tasks, as 
assessed by their 
parents/guardians 
Index: Very well, 
Moderately well, Not 
well 


Reference 


Martin et al. (2020, 
p. 16.32) 


Early numeracy 
tasks scale? 


Parents’ responses if 

their child could do the 

following when he/she 

began the first grade of 

primary/elementary 

school 

* Count by 
himself/herself 

* Recognize written 
numbers 

* Write numbers 

Response options: up to 

100 or higher, up to 20, 

up to 10, not at all 

* Do simple addition 

* Do simple subtraction 

Response options: Yes, 

no 


The higher the score 
the higher the 
student’s competency 
in performing early 
numeracy tasks, as 
assessed by their 
parents/guardians 
Index: Very well, 
Moderately well, Not 
well 


Martin et al. (2020, 
p. 16.32) 


(continued) 


The dependent variable was student achievement in mathematics and its content 
subdomains (number, measurement and geometry, and data) in TIMSS 2019. The 
following variables were used as control variables in regression models: attendance 
in preschool programs (recoded into “no attendance,” “less than three years,” and 
“three years or more”), student gender, and the TIMSS home resources for learning 


scale (Table 1). 
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Variable 


Home resources for 
learning scale* 


Description 


Based on students’ and 
parents” reports 
regarding the 
availability of five 
resources: 

* Number of books in 
the home (students) 

* Number of home 
study support 
(students) 

* Number of children's 
books in the home 
(parents) 

* Highest level of 
education of either 
parent (parents) 

* Highest level of 
occupation of either 
parent (parents) 


Information on 


Values/Response 
options 


Higher values mean 
more home resources 
Index: Many 
resources, Some 
resources, Few 
resources 


Reference 


Martin et al. (2020, 
p. 16.39) 


Fishbein et al. (2021, 


Student gender 


students' gender 
provided by students 


Boy 
Girl 


Supplement 1, p. 11) 


Attendance in 
preschool programs 


How long was your 
child visiting early 
childhood or 
pre-primary educational 
program 


Did not attend 
Less than 1 year 
1 year 

2 years 

3 years 

4 years or more 


Fishbein et al. (2021, 
Supplement 1, p. 45) 


Note *These TIMSS scales are constructed so that the scale center point of 10 is located at the mean 
score of the combined distribution of all TIMSS 2019 grade four participants. The units of the scale 
are chosen so that the standard deviation of the distribution corresponds to two scale score points 


We analyzed data using basic and advanced methods to estimate percentages, 
means, correlations, and develop regression models. We conducted all statistical 
computations using established standard procedures for data from large-scale assess- 
ments (see Sect. 5 for more details on the analysis methods and tools that we 


used). 
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4 Results 


4.1 Similarities and Differences in Students? Early Numeracy 
and Literacy Competencies Across the Dinaric Region 


We derived average scale scores for the composite variables ELT and ENT from 
the TIMSS 2019 data for the education systems from the Dinaric region (Table 2). 
According to Mullis et al. (2020), these average scale scores indicate that children 
in the Dinaric region could do, on average, most early literacy and numeracy tasks 
moderately well. 


Early Literacy Competencies Across the Dinaric Region 


In order to further explore the variation in early competencies, we investi- 
gated the percentages of students in the Dinaric region who, according to their 
parents/guardians, were able to perform specific early literacy tasks "very well" 
or "moderately well" (Fig. 1). The vast majority of parents/guardians reported that 
their child could write their name (29046 across the region), and more than 80% of 
students could recognize and write letters before attending school (with the excep- 
tion of Montenegro). Far fewer children could perform more advanced activities, like 
reading words, sentences, or even stories, or writing words. It is understandable that 
fewer students mastered skills like reading stories before starting formal schooling, 
since pre-reading skills for children aged four to five are associated with recognizing 
syllables as well as the first and last letters in a word (Cudina-Obradovié, 2002). In 
addition, children of this age group are usually emerging from the scribble stage of 
development when their writing begins to look like real letters and words, combined 
with shapes that are visually similar to displaced and valid letters (Baucal, 2012; 
Hope, 2008). 


Table2 Average scale scores for composite variables early literacy tasks (ELT) and early numeracy 
tasks (ENT) 


Education system Average score on the ELT scale Average score on the ENT 
scale 

Albania 10.7 (0.07) |10.6 (0.07) 
Bosnia & Herzegovina | 10.2 (0.03) 9.7 (0.04) 
Croatia 10.6 (0.04) |10.4 (0.05) 
Kosovo? 10.7 (0.04) | 10.5 (0.05) 
Montenegro 9.8 (0.03) 9.6 (0.03) 
North Macedonia 10.0 (0.05) | 10.2 (0.06) 
Serbia? 10.0 (0.04) |10.0 (0.05) 


Notes Standard errors appear in parentheses 
? National defined population covers 90-95% of the national target population 
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Fig. 1 Percentage of parents in the Dinaric region who reported that their child could do the 
a reading and b writing tasks “very well” or “moderately well” in response to the TIMSS 2019 
Early Learning Survey Notes In Kosovo and Serbia, the national defined population covers 90-95% 
of the national target population. In North Macedonia, data were available for >70% of students, 
but <85% of students 


Among the education systems from the Dinaric region, there was large variation 
in the percentages of parents reporting that their child was able to perform the activ- 
ities related to reading and writing words, sentences, or stories (Fig. 1). Two distinct 
groups could be distinguished among the participating entities: parents/guardians in 
Albania, Bosnia and Herzegovina, Croatia, and Kosovo tended to report that their 
children had acquired relatively high literacy competencies during the preschool 
period, especially regarding reading sentences and stories, while slightly lower 
percentages of parents/guardians in Montenegro, North Macedonia, and Serbia said 
that their children had acquired these cognitively demanding skills before entering 
school (Fig. 1). 


Early Numeracy Competencies Across the Dinaric Region 


According to their parents/guardians’ reports, few students were unable to count by 
themselves before entering school (Fig. 2). Similar results were reported for the skills 
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Fig.2 Percentage of parents reporting that their child could a count by himself/herself, b recognize 
written numbers, c write numbers, and d add or subtract numbers Note In Kosovo and Serbia, the 
national defined population covers 90-95% of the national target population 
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ofrecognizing or writing numbers, with the exception of Montenegro, which reported 
higher percentages of children entering the school system with no demonstrated 
ability torecognize and write numbers (12% and 10%,respectively). Recognizing and 
writing numbers over 20 was more challenging for preschoolers; only approximately 
a third of parents/guardians across the region reported their children had developed 
those skills. The variation among participants from the Dinaric region was also 
pronounced for these two categories (Fig. 2); for example, the results from Albania 
and Montenegro differ by approximately 20%. 

Regarding the more advanced numeracy skills, such as addition and subtraction, 
parents/guardians’ reports suggested that, on average, 79% of the children in the 
region were able to do addition, while 70% could do subtraction. However, again 
there was some variation between participating systems; for instance, there was an 
almost 20% point difference between Albania and Bosnia and Herzegovina (Fig. 2). 

For preschoolers, writing numbers is the hardest task of all listed early numeracy 
tasks. Children must be capable of reproducing a graphical representation of the 
number without fully understanding the relationship between the symbol and concept 
of quantity behind it. At the same time, the development of counting is happening 
automatically, possibly due to the high number of traditional counting rhymes that 
exist in the Dinaric region. 

An interesting finding for students from the Dinaric region is that as many chil- 
dren mastered addition as mastered counting. Acquired symbolic number, namely 
an understanding of the concept of the cardinality principle, as well as more highly 
developed counting skills (the ability to count up to 100 by the age of six) have 
been shown to be good indicators of later arithmetic skills (Góbel et al., 2014; 
Muldoon et al., 2013). It is extraordinary to find that most parents/guardians in 
the Dinaric region reported that, alongside abilities to count up to and beyond 20, 
children possessed higher or very similar values for skills related to addition. This 
suggests that children are capable of performing both activities at the same level. 
However, closer inspection of the wording of the survey question (TIMSS & PIRLS 
International Study Center, 2018) might provide another explanation; it could have 
been more specific and it did not fully define “simple addition.” Although memo- 
rization of addition expressions from children’s nurseries could be considered an 
indicator of arithmetic skills development, we note that models of numeracy devel- 
opment predict several competences/skills are developed (symbolic number system, 
estimation of quantity without counting, comparison, approximation and numerical 
magnitudes, and counting strategies) before children are able to personally imple- 
ment “simple” written calculations or other arithmetic skills (LeFevre et al., 2010; 
Siegler & Braithwaite, 2017; Von Aster & Shalev, 2007). 


4.2 Relating ELT and ENT to Mathematics Achievement 


We first determined whether the predictor variables (ELT and ENT) and achieve- 
ment were related using correlation analysis (Table 3). We found that both EL and 


110 Z. Džumhur et al. 


Table 3 Correlations between parents/guardian assessments of children's early literacy and 
numeracy competencies and their TIMSS 2019 mathematics achievement 


Education system Correlation between ELT and Correlation between ENT and 
mathematics achievement mathematics achievement 

Albania 0.30 (0.03) | 0.24 (0.03) 
Bosnia & Herzegovina | 0.19 (0.02) | 0.29 (0.02) 
Croatia 0.30 (0.02) | 0.33 (0.02) 
Kosovo? 0.15 (0.02) | 0.22 (0.02) 
Montenegro 0.22 (0.02) | 0.29 (0.02) 
North Macedonia 0.19 (0.03) | 0.29 (0.02) 
Serbia? 0.36 (0.02) | 0.40 (0.02) 


Notes ELT — early literacy tasks, ENT — early numeracy tasks. Statistically significant (p « 0.05) 
correlation coefficients are shown in bold. Standard errors appear in parentheses 
*National defined population covers 90-95% of the national target population 


EN competencies were significantly correlated with achievement in all participating 
entities. However, correlations varied considerably; Serbia had the highest correlation 
coefficient (0.36) and Kosovo the lowest coefficient (0.15). 

We applied three regression models to explore to what extent the composite vari- 
ables ENT and ELT predicted mathematics achievement (either separately and/or in 
combination), controlling for variables related to students' individual characteristics 
such as gender, home resources for learning, and preschool attendance: 


(1) Model 1 ENT (early numeracy tasks); 
(2) Model 2 ELT (early literacy tasks); 
(3) Model 3 ELT + ENT (early literacy tasks + early numeracy tasks). 


We found that these variables were significant predictors of mathematics achieve- 
ment in the Dinaric region (Table 4). This was generally true for all three models, with 
only one exception (North Macedonia, model 3). The models explained considerable 
amounts of variation in achievement; for example, in Model 1 for Albania, 19% of 
the variation in mathematics achievement of grade four students could be explained 
by the model. Positive values for standardized coefficients indicated positive rela- 
tionships between ELT/ENT and achievement. It should be noted that values for 
the variance explained (R?) varied across the region. The lowest were observed for 
Kosovo, Albania, and Bosnia and Herzegovina, and the highest for North Macedonia 
and Serbia (Table 3). In assessing the relative importance of predictor variables within 
the models, standardized regression coefficients were significant for each education 
system for models 1 and 2, but varied across the Dinaric region (Table 4). In model 
3, the regression coefficients of ENT were higher than those for ELT in Bosnia 
and Herzegovina, Kosovo, and North Macedonia, and the 9596 confidence intervals 
between the two scales did not overlap, suggesting that the coefficients for these 
participants differed. However, this did not hold in Albania, Croatia, Montenegro, 
and Serbia. This implies that, in the three systems where the regression coefficients for 
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ENT were higher, ENT was a more powerful predictor of mathematics achievement 
than ELT, when accounting for both measures. 

Our analyses indicated that the relative contributions of home resources for 
learning, student gender, and preschool attendance toward student achievement 
varied across all three models (Tables 5, 6 and 7). As expected, home resources 
were found to be a more consistent significant predictor of mathematics achieve- 
ment than the predictor variables ENT, ELT or ELT + ENT, respectively (i.e., the 
regression coefficients were larger, indicating a larger association or relationship; 
Cohen et al., 2003). The regression coefficients for home resources remained similar 
across participating systems in all our analyses, although regression coefficients were 
notably larger for North Macedonia and Serbia than for other systems. Conversely, 
gender and length of preschool attendance (<3 years, or >3 years) were less important 
contributors for all three models. 

The negative regression coefficients for gender imply that boys tended to have 
higher mathematics achievement than girls, after controlling for all other factors 
included in this model. However, our analyses revealed that gender had very 
little impact on achievement in Bosnia and Herzegovina, Croatia (Model 1-3), 
Montenegro (Models 2-3), and Serbia (Model 2), and no significant impact in other 
systems and models. Preschool attendance was also shown to be a weak predictor of 
mathematics achievement. Our research only identified significant regression coeffi- 
cients related to <3 years preschool attendance in Kosovo (Model 1-3) and in Bosnia 
and Herzegovina (Model 3). Equally, regression coefficients were low, but signif- 
icant for preschool attendance of three years or longer in only Kosovo and North 
Macedonia. 


Table 5 Standardized regression coefficients for the control variables in Model 1 (ENT) 


Education Home resources Gender (girl) Preschool Preschool 
system for learning scale attendance attendance 
(<3 years) (=3 years) 
Albania 0.36 (0.03) 
Bosnia & 0.31 (0.02) | —0.06 | (0.02) 
Herzegovina 
Croatia 0.29 (0.03) | —0.09 | (0.02) 
Kosovo? 0.26 (0.03) 0.08 (0.03) | 0.06 (0.03) 
Montenegro |0.30 (0.02) 
North 0.40 (0.03) 0.08 (0.04) 
Macedonia 
Serbia? 0.40 (0.02) 


Notes Only statistically significant (p « 0.05) regression coefficients are shown, empty cells indicate 
values were not significant. Standard errors appear in parentheses 
? National defined population covers 90-95% of the national target population 
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Table 6 Standardized regression coefficients for the control variables in Model 2 (ELT) 


Education Home resources Gender (girl) Preschool Preschool 
system for learning scale attendance attendance 
(<3 years) (73 years) 
Albania 0.34 (0.03) 
Bosnia & 0.33 (0.02) | —0.09 | (0.02) 
Herzegovina 
Croatia 0.30 (0.02) | —0.14 | (0.02) 
Kosovo? 0.26 (0.03) 0.07 (0.03) | 0.06 (0.03) 
Montenegro | 0.32 (0.02) | —0.06 | (0.02) 
North 0.42 (0.03) 0.08 (0.04) 
Macedonia 
Serbia? 0.41 (0.02) | —0.05 .|(0.02) 


Notes Only statistically significant (p « 0.05) regression coefficients are shown, empty cells indicate 
values were not significant. Standard errors appear in parentheses 
? National defined population covers 90-95% of the national target population 


Table 7 Standardized regression coefficients for the control variables in Model 3 (ENT + ELT) 


Education Home resources Gender (girl) Preschool Preschool 
system for learning scale attendance attendance 
(«3 years) (273 years) 


Albania 

Bosnia & 

Herzegovina 

Croatia 0.28 (0.03) | —0.12 | (0.02) 

Kosovo? 0.25 (0.03) 0.07 (0.03) | 0.06 (0.03) 
Montenegro | 0.30 (0.02) | —0.04 | (0.02) 

North 0.40 (0.03) 0.07 (0.04) 
Macedonia 

Serbia* 0.37 (0.02) 


Notes Only statistically significant (p < 0.05) regression coefficients are shown, empty cells indicate 
values were not significant. Standard errors appear in parentheses 
*National defined population covers 90-95% of the national target population 


The ENT Variable and Content Domains 


Given that the ENT variable was shown to be a relatively strong predictor of mathe- 
matics achievement for the Dinaric region in TIMSS 2019, we investigated the extent 
to which ENT predicated achievement could be attributed to the different content 
domains covered in TIMSS 2019 (number, measurement and geometry, and data). As 
in our previous regression models, variables related to home resources for learning, 
preschool attendance, and gender were used as controls. 
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Three regression models were used to evaluate this question with different 
dependent variables: 


(1) Model number: student achievement in the subdomain “number” predicted by 
ENT and control variables; 

(2 Model measurement and geometry: student achievement in the subdomain 
"measurement and geometry" predicted by ENT and control variables; 

(3) Model data: student achievement in the subdomain "data" predicted by ENT 
and control variables. 


We found that ENT significantly predicted achievement not only for numeracy 
tasks, but also for measurement and geometry, and data tasks (see Tables S.5, S.6 
and S.7 in the supplementary materials available for download at www.iea.nl/public 
ations/RfEVol13). As we already noted for the relation between ENT and the overall 
achievement scores in mathematics for TIMSS 2019, the proportion of variance and 
regression coefficients were highest for Serbia, followed by North Macedonia. The 
explained variance ranged from 20 to 38% in all three cognitive domains; there 
was also a high correlation for the *number" domain in Croatia and Bosnia and 
Herzegovina, explaining up to 21% of the variance. Taking control variables into 
consideration, we observed patterns that were broadly similar to those obtained for 
Models 1-3, with home resources remaining a significant factor for all three models 
and for all Dinaric TIMSS participants. Gender and preschool attendance were also 
weakly significant, but had only low impact in a few instances across the Dinaric 
region. 


4.3 Gender Differences 


Gender differences in estimates of ELT skills were significant for all participants 
from the Dinaric region in favor of girls (Fig. 3). Estimates for girls’ ELT skills 
ranged from 10.0 in Montenegro to 10.9 in Kosovo (the average estimate for girls 
in the Dinaric region was 10.5). Estimates for boys' ELT skills ranged from 9.6 in 
Montenegro to 10.6 in Albania and Kosovo (the average estimate for boys in Dinaric 
region was 10.1). 

In contrast, differences between girls and boys were smaller for ENT than for ELT 
skills. Significant differences were found in only three of the participating systems; all 
favored boys, but these differences were not as pronounced as the consistent gender 
difference observed for the ELT variable. Estimates for girls ENT skills ranged from 
9.6 in Montenegro to 10.6 in Albania (the average estimate for girls in the Dinaric 
region was 10.1). Estimates for boys' ENT skills ranged from 9.6 in Montenegro to 
10.6 in Albania (the average estimate for boys in Dinaric region was 10.2). 

When assessing the relations between ELT and ENT estimates and mathe- 
matics achievement by gender, correlation analyses showed significant, but weak 
to moderate positive associations among those variables (Table 8), indicating that 
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Fig.3 Difference in parents/guardians' assessment of their child's a preschool literacy competency 
and b preschool numeracy competency by gender Notes Differences were statistically significant 
(p « 0.05) in all participating education systems for the early literacy tasks, but only statistically 
significant in Bosnia and Herzegovina, Croatia, and Kosovo for the early numeracy tasks. In Kosovo 
and Serbia, the national defined population covers 90-95% of the national target population 


higher ELT/ENT scores were related to better achievement on average across all 
students. 

Overall, the correlation coefficients for ELT, both for girls and boys, tended to be 
lower than the correlation coefficients for ENT, but the differences were largest in 
Bosnia and Herzegovina and North Macedonia (average ELT yoy, r = 0.26, ELT ints r 
= 0.24, compared to average ENTksoys r = 0.31, ENT girls r = 0.28). For the ELT vari- 
able, two TIMSS participants attained moderate coefficients for both girls and boys 
(Serbia and Croatia). For the ENT variable, Albania, Kosovo, and North Macedonia 
had correlation coefficients of «0.3 for both boys and girls (indicating there were only 
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Table 8 Correlations between parents/guardian assessments of children's early literacy and 
numeracy competencies and their TIMSS 2019 mathematics achievement, by gender 


Education Correlation coefficient between ELT Correlation coefficient between ENT 
system and mathematics achievement and mathematics achievement 

Boys Girls Boys Girls 
Albania 0.31 (0.04) |0.29 | (0.03) | 0.28 (0.04) |0.21 | (0.03) 
Bosnia & 0.20 (0.03) | 0.19 | (0.02) | 0.30 (0.02) | 0.26 | (0.02) 
Herzegovina 
Croatia 0.32 (0.03) | 0.32 | (0.03) | 0.34 (0.03) | 0.30 | (0.03) 
Kosovo* 0.16 (0.03) | 0.14 | (0.03) | 0.23 (0.02) | 0.21 | (0.02) 
Montenegro | 0.24 (0.02) | 0.21 | (0.03) | 0.31 (0.02) | 0.27 | (0.03) 
North 0.19 (0.04) | 0.17 | (0.04) | 0.29 (0.03) | 0.29 | (0.03) 
Macedonia 
Serbia* 0.39 (0.03) | 0.34 | (0.03) | 0.41 (0.03) | 0.40 | (0.03) 


Notes All correlation coefficients were statistically significant (p < 0.05). Standard errors appear in 
parentheses 
*National defined population covers 90-95% of the national target population 


small associations between variables according to Cohen’s standard for evaluation; 
see Cohen et al., 2003), while in Croatia and Serbia, the correlation coefficients were 
>0.3 for both girls and boys (indicating only moderate correlations existed). 


5 Discussion and Conclusions 


We aimed to examine the relationship between early literacy and numerical compe- 
tencies for grade four students in the Dinaric region and their subsequent mathe- 
matics achievement in the TIMSS 2019 assessment. We found that the skills related 
to early literacy and early numeracy competencies were relatively strong predictors 
of student achievement in mathematics, in agreement with previous studies (Duncan 
et al., 2007; Moll et al., 2015), including those studies which used TIMSS data from 
earlier cycles (Soto-Calvo & Sánchez-Barrioluengo, 2016; Radišié & Seva, 2017). 
Parents/guardians from the Dinaric region generally estimated their children as 
being able to do literacy and numeracy tasks “moderately well” on the TIMSS scale 
“Could do literacy and numeracy tasks when beginning primary school” (Mullis 
et al., 2020, exhibit 5.18), although the average on this scale for the Dinaric region 
appeared to be a little higher than the EU average reported in TIMSS 2011 by Soto- 
Calvo and Sánchez-Barrioluengo's (2016) study.” The average ENT and ELT values 


2 For the participating education systems from the Dinaric region, the ELT average was 10.3 and 
the ENT average was 10.2 in TIMSS 2019. Calvo and Sánchez-Barrioluengo (2016) calculated an 
ELT average of 9.73 and an ENT average of 9.74 for EU education systems from the TIMSS 2015 
data. 
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for Croatia were similar in TIMSS 2019 to their previously reported results from 
TIMSS 2011, whereas, between TIMSS 2015 and TIMSS 2019, average ENT and 
ET values decreased in Serbia? 

However, the majority of parents/guardians reported that their children could 
recognize letters from alphabet systems used in the Dinaric region (Latin and Cyrillic 
letters), and write their own name. The most likely explanation for those results is 
that these activities are recognized as representative for school readiness, since they 
are part of assessment at the school entrance. Although writing your own name is 
one of the developmental milestones in the emergent writing system (Puranik & 
Lonigan, 2011), it is interesting that recent findings are not conclusive on whether 
this skill, represented by knowledge concerning letter names, letter sounds, or more 
complex alphabetic principles (Drouin & Harmon, 2009; Molfese et al., 2011), is to 
be considered a good indicator of children's conceptual literacy knowledge. 

When it comes to the numeracy competencies, our findings contribute toward 
understanding of the relation between early numeracy skills and different content 
domains of mathematics at school level, such as functional numeracy, geometry, data, 
or measurement. Early numeracy ability has been linked to later numeracy ability up 
tothe age of adolescence in several studies (Geary et al., 2013). Our results extend this 
further by relating early numeracy with the domains of geometry and measurement, 
and data. Results from a previous study on longitudinal predictors of mathematics 
performance (LeFevre et al., 2010) found that early numeracy skills were related to 
numeration and calculation skills, but did not find any relationship to geometry and 
measurement skills later in schooling. This apparent variation in findings may be 
attributed to the different tasks used to measure geometry and measurement compe- 
tencies. For example, TIMSS tasks related to geometry content domains involve 
measurement as well as numerical knowledge and skills (for example, students may 
be asked to calculate the perimeter of a rectangle). However, the tasks used in LeFevre 
et al. (2010) may have included a greater proportion of vocabulary items related 
to geometry, language-mediated processing on spatial arrays, and sequencing and 
patterning questions. Our findings were also broadly in agreement with numerous 
previous studies about the contribution of home resources for learning toward student 
achievement in mathematics (Cankaya & LeFevre, 2016). 

Interestingly, we found that the association between preschool attendance and 
mathematical achievement was not significant, although some previous research has 
suggested otherwise (Yoshikawa et al., 2016). This is perhaps because preschool 
education may not have a long-term effect (Magnuson et al., 2007), but it may 
also be possible that the positive effect of preschool education is more pronounced 
for children from diverse and at-risk families who attend high-quality preschool 
programs (Throndsen et al., 2020). We did not consider these aspects of preschool 
education in our analyses. As to gender effects, our results are consistent with a 
large meta-analysis by Lindberg et al. (2010), which showed that there were no 
gender differences in mathematics performance (and that male and female variances 


3 In Serbia, the average ELT value was 10.6 and average ENT was value was 10.3 for TIMSS 2015. 
In TIMSS 2019, these values had decreased to an ELT value of 10.0 and an ENT value of 10.0. 
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were closely equivalent). Given that our data was based on information collected 
from parents/guardians, we cannot dismiss the possibility that perceptions about 
the mathematical capacities and performance of some participants from the Dinaric 
region may be colored by their caregivers' beliefs and prejudices, and by established 
social gender stereotypes in the region (Steele, 1997, 2003). 

Finally, our analyses showed distinct variance among the different parental groups 
in their overall reports on children's early literacy and numeracy competencies. This 
may be attributed to the effect of age differences in sampling the Dinaric region; in 
some of the education systems, preschool refers to children of age four to five, while, 
in other systems, this could refer to children of age five to six. Observed dissimilarities 
could result from the different levels of understanding among parents/guardians 
about what constitutes early literacy and numeracy competencies and what is age 
appropriate. This lack of knowledge about developmental trajectories is recognized 
as one of the main factors related to parental misrepresentation and overestimation 
of children's capabilities (Zippert & Ramani, 2017). 

The limitations of this study are mainly related to the nature of the variables 
used in TIMSS and more generally in large-assessment studies, where the data on 
the relation between early cognitive competencies and later school achievement are 
captured only at a superficial level using few questions. In addition, when collecting 
information from parents/guardians about their children's development of mathe- 
matical competencies, parents/guardians are not always certain about placing their 
children's mathematical competencies in a comparative framework, especially when 
it comes to advanced number skills. They tend to overestimate their child's abilities 
for different number tasks, including the concept of cardinality, counting skills, and 
symbolic and non-symbolic arithmetic (Fluck et al., 2005; Zippert & Ramani, 2017). 
In TIMSS, the data on early cognitive skills of students are gathered from retrospec- 
tive parental reports (TIMSS & PIRLS International Study Center, 2018), a fact that 
should be accommodated when considering parental estimations of their children's 
EL and EN skills. 

The findings of our study suggest a range of possible avenues for researchers 
from the Dinaric region to investigate and explore in more depth, including further 
analysis of not only the early numeracy variables recognized in the literature as mean- 
ingful predictors of mathematics achievement (e.g., symbolic versus non-symbolic 
mathematical competencies) but also the measures that might provide a deeper under- 
standing of parental attitudes and practices. The findings provide a good basis for the 
creation and improvement of numeracy development programs for parents/guardians, 
and provide robust data for policymakers about the impact of early childhood math- 
ematics in current preschool curricula (Clements & Sarama, 2008; Thiel & Perry, 
2018). 

The results imply that parental estimates regarding the development of mathemat- 
ical competencies before entering school are culturally conditioned. Traditionally, 
parents/guardians in the Dinaric region believe that children are supposed to know 
how to write and read words as well as perform simple addition before entering first 
grade. Results from TIMSS 2019 show that, in Dinaric region, parents/guardians still 
have relatively high expectations when it comes to their children's early numeracy 
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skills. However, small differences across the region are noticeable, possibly due to a 
shift in parental beliefs, attitudes, and practices related to this topic. Future TIMSS 
cycles will enable researchers and policymakers to identify potential changes and 
to develop appropriate programs for parents/guardians, helping them to facilitate 
numeracy development in their children in more effective and age-appropriate ways. 
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Abstract International large-scale assessments can play a critical role in identi- 
fying factors that have an effect on student learning and achievement. IEA’s Trends 
in International Mathematics and Science Study (TIMSS), as the only international 
study of primary level mathematics and science education, is Increasingly important 
in supporting continuous improvement in the quality of education and education 
systems. TIMSS also collects background information about the material and non- 
material factors that potentially affect teaching and learning processes, and the 2019 
cycle of TIMSS provided a unique opportunity to analyze the role these factors play 
in education across the Dinaric region. Previous research has suggested that there 
are two especially important sets of socioeconomic background variables that need 
to be taken into consideration when analyzing possible factors related to student 
achievement and their attitudes toward teaching and learning at school. These are, 
firstly, personal student characteristics and their home resources and, secondly, school 
climate and material resources. Modeling of the TIMSS 2019 data for the Dinaric 
education systems indicated that material, environmental, and school climate factors 
were only weakly associated with student achievement across the region, explaining 
less than 12% of the variance in student achievement in science and less than 11% 
of the variance in mathematics achievement. These results indicate that education 
authorities in the region should not automatically assume that the material charac- 
teristics of the school environment, as well as elements of school climate, are the 
best or only areas for potential improvement. Access to home learning resources, 
parental support, and students’ and teachers’ attitudes toward learning and teaching 
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seem to be more important factors in explaining differences in student achievement 
across the Dinaric region than previously perceived. 


Keywords Bullying - Safety - School climate * School environment * School 
material resources * Trends in International Mathematics and Science Study 
(TIMSS) 


1 Introduction 


Around the world, education authorities are Interested in identifying factors that have 
an effect on their students’ achievement, instigating educational reforms that enhance 
positive elements of their systems and diminish any negative effects. International 
large-scale assessments (ILSAs) are viewed as increasingly important in supporting 
continuous improvement in the quality of education and education systems. Such 
worldwide assessments, like those conducted by the International Association for the 
Evaluation of Educational Achievement (IEA) and the Organisation for Economic 
Cooperation and Development (OECD), as well as others emerging from European 
Union (EU) initiatives, report the influence of material and non-material factors 
on teaching and learning processes. IEA's Trends in International Mathematics and 
Science Study (TIMSS) is especially important for science, technology, engineering, 
and mathematics (STEM) because it is the only international study of those subjects 
at the primary school level. 

In general, TIMSS has shown that student achievement around the globe has 
improved since the study began collecting data and measuring trends in 1995, with 
many national systems showing increasing achievement at both grades four and eight 
for the mathematics and science subjects. As Mullis said in 2016: “The positive 
trends indicate education is improving worldwide, and it's not at the expense of 
equity between high and low achieving students" (TIMSS PIRLS International Study 
Center, 2016). With this in mind, we investigated school resources and characteristics 
of the school environment across the Dinaric region; our aim was to understand what 
underlying factors promote schools as good, successful, and open places for teaching 
and learning. 

Seven participants from the Dinaric region took part in TIMSS 2019, namely 
Albania, Bosnia and Herzegovina, Croatia, Kosovo, ! Montenegro, North Macedonia, 
and Serbia. Croatia and Serbia both also participated in TIMSS 2015 (where they 
tested grade four students).” Both achieved results above the international average in 


' This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 
(United Nations 1999) and the International Court of Justice (ICJ) Opinion on the Kosovo declaration 
of independence (ICJ 2010). 

? Slovenia also participated in TIMSS 2015 (and achieved above the TIMSS international average 
results in mathematics and science), but did not participate in TIMSS 2019 survey and thus could 
not take part in this comparative analysis of the Dinaric region. 
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TIMSS 2015 for grade four science, with Serbia also scoring above the TIMSS inter- 
national average in grade four mathematics (Croatian student achievement for grade 
four mathematics was around the TIMSS international average). Both education 
systems also reported an Increase in student achievement in mathematics and science 
between TIMSS 2011 and TIMSS 2015, mirroring the global trend of improvement 
in student achievement in the subjects assessed by TIMSS. However, while this 
improvement continued for mathematics in the 2019 cycle of TIMSS in Croatia, 
a decline in science achievement was noted (despite still scoring above the TIMSS 
internationalaverage). Meanwhile, in Serbia, both assessment areas showed a decline 
in student achievement between TIMSS 2015 and TIMSS 2019, and the student 
mathematics achievement score declined below their TIMSS 2011 score (although 
still remaining above the TIMSS international average). All other Dinaric systems 
represented in this report recorded grade four student achievement scores below the 
TIMSS 2019 international average; among this group, Albania’s results were closest 
to the TIMSS 2019 international average and Kosovo’s results furthest from the 
TIMSS 2019 international average for both the mathematics and science assessment 
areas. 

We were interested in whether available school resources, the school environment, 
and school climate could be linked to student achievement in the Dinaric region. 
Prior research (Kutsyuruba et al., 2015) has indicated that these factors may play an 
important part in developing successful schools and students, but, given that cultural 
factors may also be involved, the data collected by TIMSS 2019 provides the first 
opportunity to establish the interacting associations between these factors and student 
achievement across the Dinaric region. 

For the Dinaric participants that were involved in TIMSS 2015 and earlier cycles, 
there has already been an initial exploration of these concepts and their potential 
effect. School principals reported that almost three-quarters of all students partici- 
pating in TIMSS 2015 were “affected” or “affected a lot” by the shortage of resources 
for mathematics and science instruction. In the Dinaric region at that time, 18% 
of Croatian schools reported “more than 25% students coming from economically 
disadvantaged homes (and not more than 25% from economically affluent homes);” 
in Slovenia this figure was 23%, and it was 44% in Serbia. For all three partici- 
pants, better achievement results were noted for students in schools where “more 
than 25% of the student body comes from economical affluent homes (and not more 
than 25% from economically disadvantaged homes)” than for students in schools 
in that fell into the other two groups, which contained proportionally more students 
from disadvantaged homes (Martin et al., 2016; Mullis et al., 2016a). Almost a fifth 
of primary schools in Croatia, a quarter of primary schools in Slovenia, and half of 
the primary schools in Serbia contained students from homes with (relatively) harsh 
socioeconomic conditions, and it is perhaps not unexpected that this would have 
a negative effect on learning and teaching in these schools. Many ILSAs, such as 
OECD’s Programme for International Student Assessment PISA and IEA's TIMSS 
and Progress in International Reading Literacy Study (PIRLS), have highlighted the 
importance of home environment in supporting student success (Martin et al., 2016; 
Mullis et al., 2016a, 2017, 2020; OECD, 2019c). 
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Similarly, teachers surveyed in TIMSS 2015 reported having “moderate to severe 
problems” with school conditions and resources for 17% of students in Slovenia, 
23% of students in Croatia and 35% of students in Serbia. It is interesting to note 
that, in all three participants, students from schools that teachers had identified as 
strongly affected by such problems nevertheless tended to record higher average 
achievement in mathematics and science than less affected students. The TIMSS 2015 
international results indicated that, generally, students with teachers who reported 
that their school had no problems with resources had the highest achievement, and 
students with teachers who reported that their school was “affected a lot” by problems 
with conditions and resources had the lowest average achievement among their peers 
(Martin et al., 2016; Mullis et al., 20162), which seems more in line with expectations. 
To explain the apparent deviation in the relationship between material resources at 
school and achievement in the Dinaric region, some research has suggested that, in 
conditions when material resources are lacking, teachers (and other staff) tend to 
give more attention to students' learning and are more available and willing to help 
as a form of compensation (OECD (20192). 

Another general conclusion from TIMSS 2015 was that according to parents, 
principals, and teachers, as well as students themselves, the majority of grade four 
students were attending good schools. On average, across all TIMSS 2015 partici- 
pants, 58% of parents were reportedly very satisfied with students’ school perfor- 
mance, 52% of teachers were very satisfied with their jobs, more than half the teachers 
and principals reported that their school achieved a high level of academic success 
or that there was very strong emphasis on academic success in their school (26090), 
and the majority of students (66%) reported a strong sense of school belonging. In 
the Dinaric region, the patterns found followed these general conclusions (Martin 
et al., 2016; Mullis et al., 20162). 

In TIMSS 2015, school climate was represented by a composite TIMSS “Safe 
and Orderly School" scale (Martin et al., 2016; Mullis et al., 2016a). In general, 
TIMSS 2015 found that the majority of grade four students were in safe school envi- 
ronments (56%, according to teachers) and, according to principals, 59% of schools 
had "hardly any discipline problems." Conversely, 16% of all students reported that 
they were bullied about once a week in their schools, which perhaps challenges 
teachers' and principals” generally positive perceptions of school safety and school 
climate. The percentage of students that reported being bullied in TIMSS 2015 was 
close to the TIMSS international average in Slovenia (14%), but below the TIMSS 
international average in Croatia and Serbia (896). In TIMSS 2015, 76% of students 
in Croatia attended schools where hardly any discipline problems were reported by 
their principals. Principals in Serbia and Slovenia were more critical than principals 
in Croatia about the state in their schools (they reported that while around 50% of 
students were in schools with “hardly any problems", more than one third of them 
were in schools with “minor problems"). When teachers were asked to assess safety 
and order in their schools, they were more cautious than principals in their assess- 
ment, with around half reporting that students were in *very safe and orderly schools" 
in Croatia (4896) and Serbia (5296), while Slovenian teachers were more critical in 
their assessment (around 2996). In Serbia and Slovenia, students belonging to the 
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schools that teachers reported as being very safe and orderly also tended to achieve 
the highest scores in mathematics in science. In Croatia, there was no significant 
difference in the achievement between the groups. 

Almosthalf of the students in Serbia (49%), and more than a half of the students in 
Croatia (61%) and Slovenia (82%) had teachers reporting that teaching mathematics 
and science was somewhat or very limited by student needs (Martin et al., 2016; 
Mullis et al., 2016a). In Croatia and Slovenia, students whose teachers reported that 
teaching was not at all limited achieved the best scores in mathematics. This was 
also true for the science achievement results for Slovenia, but it was interesting 
that students whose teachers reported that teaching was somewhat or very limited by 
students needs only scored a few points less on the TIMSS achievement scale. Serbia’s 
results were quite different, and students whose teachers reported that teaching was 
very limited by students needs tended to achieve the best scores in both mathematics 
and science. The TIMSS 2019 data showed similar patterns for Croatia and Serbia 
(Mullis et al., 2020). 

While it is important to assess conditions in schools, as a source of material and 
environmental support to promote student learning, a student’s home resources for 
learning (both in terms of material assets and cultural capital) are well-proven indi- 
cators of student success in school (Matkovié et al., 2019; Meinck et al., 2018). In 
TIMSS 2015, students whose parents reported many home resources for learning 
had much higher achievement than students whose parents reported some or few 
resources. The difference in achievement between the students with many home 
resources (17-18%) and those with few resources (8-996) was 142 points for math- 
ematics and 141 for science. A similarly massive difference was reported by PIRLS 
2016, and, in both TIMSS 2015 and PIRLS 2016, students whose parents reported 
often spending time with their children on early literacy and numeracy learning activ- 
ities had a higher achievement than students whose parents did so only sometimes 
or almost never (Mullis et al., 2017). 

The conceptual model of effective schools within the PIRLS and TIMSS studies 
was also put to test. An effective school was perceived as safe and orderly, 
had adequate facilities and equipment and well-resourced classrooms, was staffed 
with well-prepared teachers, it supported academic success, and provided effec- 
tive instruction. Martin and Mullis (2013, p. 8) concluded, "After controlling for 
home background, of the school environment variables, Schools Are Safe and 
Orderly was related to higher achievement in at least one subject in 15 coun- 
tries, and Schools Support Academic Success in 10 countries. Students Engaged in 
Reading, Mathematics, and Science Lessons was the most powerful school instruc- 
tion variable, related to higher achievement in at least one subject in 15 countries, 
again after controlling for home background. All in all, a school that was safe and 
orderly, promoted academic excellence, and provided engaging instruction, could be 
considered to have several important characteristics for effectiveness." 

Resources for education are generally focused on physical conditions for 
schooling, such as having enough space for classes, and ensuring basic utilities and 
perhaps specialized classrooms are available. More recent discussion on material 
resources in schools often refers exclusively to the availability of information and 
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communication technologies (ICT) in schools, namely whether students have access 
to equipment such as laptops, tablets, broadband internet, interactive classrooms, 
and e-libraries. Both of these aspects are addressed in the TIMSS background ques- 
tionnaires (TIMSS & PIRLS International Study Center 2018). Digital skills have 
been noted as being increasingly important in almost all aspects of teaching and 
learning, in acknowledgment of the need to prepare today’s students to function as 
tomorrow's digital workers (Fraillon et al., 2020).? The integration of ICT is brings 
some new innovative forms of teaching in classrooms all over the world, having both 
advantages and disadvantages (Eickelmann, 2011). 

OECD’s PISA also researches the relation between student achievement and mate- 
rial investments in education, and has repeatedly concluded that investing in the 
school system initially has positive effects on achievement, but a point is eventually 
reached when additional investments have a more modest effect on student results and 
other factors become more important. Essentially, when everything material has been 
resolved, less tangible elements of the quality of processes of teaching and learning 
will still need to be tackled to achieve more advanced results. Nevertheless, there are 
always exceptions, as OECD (2019a, p. 56) noted, “While an inadequately resourced 
education system cannot deliver good results, Estonia, with a level of expenditure 
on education that is about 30% lower than the OECD average, is nevertheless one of 
the top-performing OECD countries in reading, mathematics and science.” 

When international large-scale assessments deliver their results, additional 
research on available data is performed in almost every country around the world. In 
Croatia, PISA 2006 data showed that home socioeconomic indicators, along with the 
region of residence, explained 24% of the variance in students’ science achievement 
and confirmed how important these factors are for student achievement (Gregurovié 
& Kuti, 2010). As PISA only tests students aged 15, more information is needed 
at other school levels to make informed decisions about schooling. Reflecting on 
the results from international data prompts at least two questions about the relation- 
ship between material resources available to students and their success measured in 
terms of knowledge attainment in important learning areas. First, can provision of 
resources in school overcome the lack of resources at the individual (student, home) 
level? Secondly, can school characteristics, such as open school climate or a positive 
school culture oriented towards achievement and academic belonging, overcome a 
lack of material resources both on the individual and school level? 

In general, previous studies have established more indicative connections between 
student achievement and school environments and school climate (Bear et al., 2014; 
OECD, 2019b; Schulz et al., 2010), than between student achievement and school 
material resources. For instance, TIMSS 2015 results have shown that, for almost 
all grade four students, a positive sense of school belonging was related to higher 
average mathematics and science achievement (Martin et al., 2016; Mullis et al., 
20162). 


? Here the term "digital" does not simply refer to digital machines and processes, but to the entire 
political, social, and economic context and infrastructure within which they have emerged. We now 
live in a "digital age" (Burston et al., 2010, p. 215). 
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Having in mind that one of the most important goals of every teaching process 
is to help students become future prosperous adults by putting emphasis on both 
cognitive outcomes and affective dimensions (attitudes, values, and beliefs), educa- 
tional systems that aim to be successful should go beyond procurement of material 
resources. Investing in the continuous professional development of teachers and prin- 
cipals is commonly recommended as a means of ensuring quality education, but other 
recommendations include investment in developing transversal (lifelong learning) 
skills or widening use of ICT in school (Drigas & Vasiliki, 2015; OECD, 2019d; 
UNESCO [United Nations Educational, Scientific and Cultural Organization], 2014; 
Webb & Cox, 2004). 


2 Methodology and Research Questions 


We aimed to investigate whether a particular set of contextual factors was related to 
achievement, and if and to what extent these factors represented important elements 
of school life. Our research was designed to address the relative importance of two 
factors that previous research has suggested may be associated with student achieve- 
ment. Firstly, how important were school material resources and the school physical 
environment (in terms of general wealth or plurality of school possessions, i.e., impor- 
tant school equipment and spaces or lack of thereof), school location, and principals’ 
perceptions of the affluence of the families from which enrolled students come from. 
Secondly, how important was the overall school climate? The elements of school 
climate here include the social determinants of everyday school life, such as student 
issues that affect teaching, safe and orderly school environments (as reported by 
teachers), and bullying among students (as reported by students). 
From this we distil three critical research questions: 


(1) How well equipped with material resources for learning are schools across the 
Dinaric region? 

(2) What can TIMSS tell us about the learning environment in schools across the 
Dinaric region? 

(3) How comparable are important aspects of school climate across the Dinaric 
region? 


We used data collected by TIMSS 2019 from seven educational systems across the 
Dinaric region in our analyses. These included students’ achievement results at grade 
four in mathematics and science, and contextual information derived from responses 
to the students’, teachers’ and principals’ questionnaires. For more information about 
samples, methods, procedures, and data that we used, see Sect. 5 and the TIMSS 2019 
technical report (Martin et al., 2020). 
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2.1 Indicators and Variables Used 


We identified several variables and scales in the TIMSS 2019 international reports 
as being of potential interest for our research (Table 1). We investigated one of 
the main aspects of schooling by creating two indexes to assess the availability 
of material resources in schools, one for mathematics and one for science. These 
indexes combined teachers’ and principals’ responses to questions about whether 
the school possessed a number of specific items (such as computers or a library) and 
the prevalence of different conditions posing obstacles for teaching into a simple 
summative “Index of School Material Resources” (see Table 1 and Tables S.8 and 
S.9 in the supplementary materials available for download at www.iea.nl/publicati 
ons/RfEVol13). 

The Index of School Material Resources combines information collected by 
TIMSS 2019 on the availability of computers during mathematics/science lessons, 
existence and size of the school library, existence of classroom libraries, provision 
of digital learning resources, and instruction being affected by mathematics/science 
lessons resource shortages. The Index of School Material Resources for teaching 
science comprised one additional variable about the availability of a dedicated science 
laboratory in the school. For both mathematics and science, we split the derived index 
into three categories: (1) few resources available, (2) some resources available, and 
(3) many resources available in the school (see Table S.10 in the supplementary 
materials available for download at www.iea.nl/publications/RfEVol13). 

Among the contextual data TIMSS collects, there are several indicators regarding 
the school environment. In the school questionnaire, principals were asked whether 
the school is situated in an urban or rural settlement and about student composition in 
their school (if more students come from disadvantaged homes or more students come 
from affluent home backgrounds). We analyzed the relationship between student 
achievement and the factors creating the school environment (whether the school 
was located in an urban or rural environment and the school principal's assessment 
of the school composition). These demographic determinants have been of interest to 
researchers for decades, in their attempts to define what conditions underlie student 
achievement; higher student achievement has been linked to urban and/or wealthier 
environments (see chapter “Scaffolding the Learning in Rural and Urban Schools: 
Similarities and Differences" for more information on this topic). 

The third factor that we addressed was school climate, which we reduced to the 
aspect of perceptions of safety and order within school. Defining school climate 
is complex, despite often being cited as an important explanatory factor for many 
student outcomes (Brand et al., 2008; Cohen et al., 2009; Hoy et al., 1991). TIMSS 
reports have consistently shown a positive relationship between student achievement 
and teachers' and principals” reports that the school is safe and orderly (Martin 
et al., 2016; Mullis et al., 2016a, 2020). The TIMSS scale on student bullying in 
school, reported by students themselves, is also important element of assessing the 
overall safety and state of interrelations within the school and thus included into 
this analysis (Martin et al., 2020). In TIMSS frameworks bullying is defined as 


131 


The Role of Learning Resources, School Environment, and Climate ... 


(pənunuoo) 


8cror d 
*(OcOC) ‘Te 19 UIE 


Jo V (£) 
euis (c) 


epi Ain (D 
18911039370 OMY) YM Xəpur 
Uv OJUT DopIAIp SI 9[eos ML 


JOT ? ‘awos 

qe æ Jou :suondo asuodsay 
SANGUE juopnis 1u3rə 

Aq Dam st sse[o po[dures 

eu) ur Surqoeo) Aa) qonur Aou 
UO sə)gurnsə ,SIoQ2v9) JO ALIŞ 


ALS uononnsur 
Joy Ápeor jou sjuopms A0 
Dot Surqoeo) uioodssv[.) 


q6'S FAIL Pur q8'S NWEL 
aoUaIojoy 


sooInosa1 MÆ (€) 
səoSInosəi 20106 (7) 
soomosar Aue (T) 
:S9r1039jeo IAI) ur xopug 


suondo esuodsoj/sonjvA 


(ATWO oouoros 

10J XOpUT) Á1oje1oqe[ o2uoIog e 
uononnsur 3unoəəje 
suossə| songuioqyeur 

10J sə21nosə: Jo IFLENOYŞ « 
səomosər 

SUIUILI] [0)ISID JO UOISTAOIG e 
SOLIeIql] 

UIOOISSU[O JO Ə2UƏ1]SIX$ e 
Araq 

Ioouos Jo ƏZIS pue oouojsIXq < 
suossə| sonpurəou)ptu 

SULINP o[qe[reA? s1ojnduio;) + 

sgoInOsal AAY JO ÁjypIqe[reAe 

ou Surpiesor sodar ,s1oqoeo] 

pue ,spedrouud uo poseq xopug 


uonduosoeq 


SOOINOSIY 


IEH [OOYDS Jo xepuy | saomnosal JENOU [oouos 


e[eos/so[Qe IVA. podsy 


sosÁqeue əu ur pəsn SALIS pue gatdptUpaA ou) JOIST] T ABL 


I. Elezovié et al. 


132 


(pənunuoo) 


61 `d ‘g yuəurə|ddnç 
*(1Z02) "Ie 19 urəqusiq 


səurou 

quonyye A[[e2rurouooo 

UIOJJ %CZ uuu) 9JOUI Jou 

pue səutou poSvjuvApesip 

K[[eorurouooo (OU souioo 

Apog 1uəpms ou Jo %çZ 

UU) IIOU Ə1Ə9uA S[OOYIS 
:peSejueApesrp o10JA (€) 

SUOTJEUIQUIOO 

asuodsal o[qissod 19470 

IY :posejueapesip o1our 
JOU juoanpjg? 21001 YNN (C) 
səurou poSejueApesIp 
K[[eorurouooeo WOU % GZ uey} 
IOW jou pue səurou juan[pge 
K[[eorurouooo uioJj seuroo Ápoq 
juopnis au Jo % ÇT UE) ATOU 
91ouA^ STOOYIS UMY AOW 
(T) :seuosogeo ooi UNA xopuj 


200€ 

uet o1our *o506-9c *o5ec- TI 

*950 1—0 :suondo asuodsay 

səuiou juengge Aj[eorurouooq e 
souioq 

posejueapesip Á[[eorurouooq e 

WOT, Suruioo 

Jooyos ou) ur sjuopnis Jo 

areys oy} uo yodat s,[edrourq 


punoJSxoeq orurouooooroos 
Áq uonisoduioo [oouyog 


JuəuruorTAu9 [ooqog 


T8 pue LL "dd ‘q juourojddng 
*(1c0c) Te 19 urequsr 
oouol9Jow 


ON ©) 


89A (D 
"mat 19d suondo 


asuodsal OM) 310M INUL 


suondo əsuodsəs/sənje) 


səuməuios osn ULI sse[o 

yeu) siə)nduro2 sey |oouos e 
IWYS ULI sjuopnis 

jeu s1ojnduioo sey sse « 

Jaynduros sey juopnis yeg . 

Sat 

oo1g o) sasuodsar ,s1oqoeo], 


uonduoseq 


Suossə| 22uo1os 10J s1o3nduroo 
Jo Knprqe[rAy/suosso[ 
someuioueur 

10} s1ojnduroo jo Áj[rqereAV 


e[eos/so[Qe IVA. 


jedsy 


(penunuoo) T AEL 


133 


The Role of Learning Resources, School Environment, and Climate ... 


€ TTOA H3s]/suoneorqnd/pueor MMM Je peo|uAop 1oJ o[qe[reAe serou ejuouro[ddns əy} 1T[nsuoo oseo[d ee pue ge SALL Jod 


(OZOZ) utəqusr, pue UTA oos eseojd *uononusuoo o[eos uo 


UONVULIOJUT 9JOUI 104 `sJutod 3109S ILIS oA 0} spuodsoxroo uonnqrnsrp og JO uonerAop prepuejs ou) WY} os uosoqo ore AVIS ou Jo sirun ou p ^sjuedronied mot 
PLIS 610c SSIALL II? Jo uonnqugsrp peurquioo oq jo e109s ueoui OY) 3? PALIO st 0[ JO 1urod 1ojuoo PLIS oq jeu) OS pojonjsuoo I SAVIS SSINLL ƏSƏU L, $27/0N 


69°91 `d '(QcOc) Te 19 un1e]q 


Kp[oo^ 1sout[v (€) 


ÁAppuour soupy (c) 
JƏAƏU ]SOUI[E IO I9A9ON| 


(D) :seuosogeo oor YIM xopur 
up olut poplAIp SI o[e2s ou], 


IeoÁ [ooqos juaxmo 

ey) SULINp [oouos æ SurAq[[nq 
Surououedxo jo Aouanbary am 
jnoqe sjueurojejs UIA YM 
Juəurəər8p sjuopnis Jo o[vog 


eSurcT[Ng juopnis 


prror d 
‘(OZOZ) ‘Te 39 UIE 


K[1op1o pue ayes up sso (€) 
Kprop1o pue əjes wymowos (2) 
Kpropio pue ayes AIBA (T) 
:SOLI0S9}V9 IA YIM xəpur 

up olut PIPIAIP SI OLIS Əu L 


Joouos 
ou 1noqe sjuourojuis IYF YM 
juouroolgv ,SIOYORI} JO BBS 


sloouos Ápropio pue avg 


auno [oouos 


€6 `d ‘q 1ueuro[ddng 
*(1c0c) Te 19 urequsrq 
oouol9Joy 


(pena 
Jowa UMO} pews) emy (7) 
(Qo umipəur 
*uequnqns ‘ueqgin) ueqgi() (1) 
:S11039]89 OM} Olut 
pəpooƏ: se^ uorje»o[ Jooos 


suondo osuodsoj/sonjvA 


[eins gour 
‘OSETIA 10 UMO) [[EUIS ‘UMO} 
o3] 10 ÁV ezis Wintpaw ‘(Rare 
ueqin Jo sjirysjno Jo ə8uur uo) 
ueqingns *(peyepndod ATasuap) 
ueqin :suondo asuodsay 
¿DƏ1eoo[ SI [oouos 1noÁ 

WoIQA UI eow ojerpouirui ou 
soquosop 1səq gorqA :uonsonb 
eu) o1 sesuodsoi ,spedrouuq 


uonduoseq 


uoneoo[ Jooos 


e[eos/so[Qe IVA. 


jedsy 


(penunuoo) I əlqeL 


134 L. Elezovié et al. 


“repeated aggressive behavior that is intended to harm students who are physically 
or psychologically less strong, and takes a variety of forms ranging from name calling 
to inflicting mental and physical harm” (Mullis and Martin 2017, p. 68). For some, this 
may be perceived as narrowed perspective of the concept of school climate, which is 
why we chose to analyze both the physical and social dimensions of school life in an 
attempt to provide a multidimensional approach. We thus undertook a comparative 
analysis of teachers’ perspectives on safety and order at school and students’ reports 
on bullying (aggregated at school level). As many national authorities around the 
world are aware, and the TIMSS 2019 international report reconfirmed (Mullis et al., 
2020), the question of school safety (i.e., student bullying) remains an important 
problem in education. The teacher Safe and Orderly School scale encompasses of 
eight statements: one asking about conditions outside of the school (i.e., safety in the 
neighborhood), three about teachers' subjective feeling of safety and order within the 
school, and another three about students' adherence to school discipline (respecting 
the rules, teachers and property). We categorized students as being in *very safe and 
orderly schools" if, on average, their teachers agreed a lot with four of the eight 
statements and agreed a little with the other four statements. 

Another indicator that we used to assess school climate was the TIMSS 2019 
scale named “Classroom Teaching Limited by Students Not Ready for Instruction,” 
which is composed of eight variables collected by the TIMSS teacher questionnaire. 
These questions assess teachers' perceptions of the severity of different limitations 
that negatively affect their classes. Teachers were asked whether their students lacked 
prerequisite knowledge or skills, suffered from lack of basic nutrition or not enough 
sleep, were absent from class, disruptive or disinterested, had to deal impairments 
(either mental, emotional or psychological), or did not understand the language of 
instruction. 

We used these variables as predictors in regression analyses that investigated 
whether those elements of school life were related to student achievement. 


3 Results 


3.1 Material Resources for Learning in Schools 


3.1. Index of School Material Resources 


As explained in Sect. 2.1, we created two indexes to explore the effects of school 
material resources, one for mathematics and one for science; the science material 
resources index contained one additional variable (availability of a science labora- 
tory in the school). Not having a science laboratory in school was related to lower 
achievement results in science only in one system (Albania). In Montenegro, there 
was no difference in science achievement among the students in schools with or 
without a science laboratory, and in Bosnia and Herzegovina, Croatia, and Kosovo, 
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the difference was small and insignificant. In Montenegro and Serbia, students from 
schools without science laboratories achieved higher scores in science (more than 
10 points higher on average) than those in schools with a science laboratory. 

We further examined principals” reports about conditions for teaching related 
to shortage of resources. Across the Dinaric region, relatively few students were 
affected either “somewhat” or “a lot” by shortages of resources for mathematics and 
science instruction, with the lowest percentages reported in Kosovo, Albania, and 
North Macedonia (<8%), and the highest percentage of affected students in Serbia 
(20%). 

These results seem to differ from teachers” reports; this may be because principals 
are either less aware of the resource problems reported by their teachers or less 
willing to admit classroom resource issues. The distribution of material resources for 
mathematics lessons varies significantly across the region (Fig. 1). Data from Bosnia 
and Herzegovina, Croatia, Montenegro, North Macedonia, and Serbia were quite 
consistent, with more than half to two-thirds of students belonging to the intermediate 
category that enjoys “some resources” (from 57% in North Macedonia to 76% in 
Croatia). In Albania and Kosovo, however, almost two-thirds of all students attended 
schools where principals indicated that their school was equipped with comparatively 
few resources. Only six percent of students in Albania were reported as having 
“many resources” and, in Kosovo, no students fell into this category. In interpreting 
these statistics, it is important to note that our school material resources scale and/or 
constructed index was comprised of physical objects and spaces, while, in the TIMSS 
2019 schools questionnaire, principals responded to questions on shortages directly 
aimed at identifying specific issues, such as providing contents and tools that assisted 
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Fig.1 Index of School Material Resources for Mathematics. Percentage of students in schools with 
different amounts of resources for mathematics lessons. Note In Kosovo and Serbia, the national 
defined population covers 90-95% of the national target population 
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different amounts of resources for science lessons. Note In Kosovo and Serbia, the national defined 
population covers 90-95% of the national target population 


teaching, along with questions about the availability of specialized staff (teachers); 
this last question was of particular interest to STEM education in the Dinaric systems. 

The distribution of material resources for science lessons was very similar (Fig. 2). 
In Montenegro, North Macedonia, and Serbia, principals’ reports indicated that 
around half the students belonged to the intermediate category of “some resources”. 
Around two-thirds of students were in this category in Bosnia and Herzegovina (64%) 
and Croatia (72%), and around a third in Albania (31%) and Kosovo (36%). Croatia 
and Serbia had the smallest number of students in the category with “few resources” 
(5% and 8%, respectively), while Albania and Kosovo had the smallest number of 
students in the category of “many resources” (10% and 7%, respectively). 

Using the Index of School Material Resources, we found that, in three of the 
Dinaric participants, differences in mathematics achievement among students at 
schools were related to the amount of resources. In Albania, on average, students 
at schools with some resources scored 40 points more than students at schools with 
only few resources, and students at schools with many resources scored, on average, 
73 points more than their peers at schools with few resources. In Croatia, students at 
schools with few resources, on average, scored 25 points less on the mathematics scale 
than students at schools with some or many resources. In Serbia, there was a 35 point 
achievement gap between students at schools with low resources and those at schools 
with many resources. However, we found no significant similar achievement gaps 
in Bosnia and Herzegovina, Kosovo, Montenegro, and North Macedonia. Regarding 
science achievement, we found similarly that students at schools with more resources 
on average tended to score higher on the TIMSS assessment, except in Montenegro; 
however, the achievement gap was only significant in Albania (Fig. 3). 
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Fig.3 Difference in a mean mathematics achievement and b mean science achievement between 
TIMSS achievement scores for students at schools with many resources and students at schools with 
few resources. Notes *The difference is statistically significant (p « 0.05). In Kosovo and Serbia, 
the national defined population covers 90-95% of the national target population 


3.1.2 Information and Communication Technology Resources 


We found that, on average across the Dinaric region, most students were in schools 
that were equipped with computers for class use, with the highest percentages in 
Croatia (97% both for mathematics and science lessons) and the lowest in Kosovo 
(54% for mathematics lessons) and Macedonia (63% for science lessons) (Fig. 4). 
When teachers were asked whether each student had a computer to use in mathematics 
and/or science classes, the situation differed; the highest percentages were in Bosnia 
and Herzegovina (3696 for mathematics and 30% for science), and lowest in Kosovo 
and Serbia (<3%). The computer-student ratio ranged widely across the region, from 
0.14 in Albania and Kosovo, 0.22 in Serbia, 0.24 in Croatia, 0.25 in Montenegro, 
and 0.41 in Bosnia and Herzegovina, to 0.77 in North Macedonia. 

As well as providing hardware, there is a more sophisticated aspect to ICT in 
schools, reflected by the construction of online networks through interactive tools 
and the publication of online content for teaching and learning, such as providing 
digital learning resources. The progress toward full integration of ICT into teaching 
and learning has been largely gradual up until 2020, when the COVID-19 pandemic 
threw education systems around the world into “overnight” digitalization, whether 
they were prepared for itor not. Across the Dinaric region, TIMSS 2019 data indicated 
that the provision of "online learning management systems" differed substantially 
(Table 2). Principals reported that students' access to digital learning resources was 
good (Table 2). 
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Fig. A Student access to computers in school for mathematics and science lessons: a percentage of 
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class use. Note In Kosovo and Serbia the national defined population covers 90-95% of the national 
target population. 


3.2 School Environments Across Dinaric Countries 


In terms of school location, more than half of the students are located in urban 
areas in all seven participants, with the highest percentage in Montenegro (85%) 
and the lowest percentages in Croatia and Kosovo (57%). In general, more students 
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Table 2 Principals’ reports of access to digital resources in TIMSS 2019 


Education system Schools have access to online Students have access to digital 
learning management systems (%) | learning resources (%) 

Albania 15 (2.9) | 26 (3.8) 
Bosnia & Herzegovina | 27 (3.3) | 47 (4.2) 
Croatia 50 (4.3) | 80 (3.6) 
Kosovo? 13 (3.0) |31 (4.3) 
Montenegro 46 (0.5) | 63 (0.5) 
North Macedonia 62 (4.6) | 68 (3.8) 
Serbia* 71 (3.5) | 76 (3.4) 


Notes Standard errors appear in parentheses. 
*National defined population covers 90-95% of the national target population 


attend urban schools, and more disadvantaged students tend to attend schools situated 
in rural areas (see chapter “Scaffolding the Learning in Rural and Urban Schools: 
Similarities and Differences” for a more detailed analysis of this topic). 

According to their principals, the percentage of students at more disadvantaged 
schools ranged from 13% in Croatia to 42% in Albania. Principals in North Mace- 
donia reported that 66% of students were in more affluent schools; this was the 
highest perceived percentage for that category in the Dinaric region. 

Previous research (Mullis et al., 2016a; OECD, 2019a) has shown that student 
achievement in mathematics is related to student home socioeconomic status or 
school principals’ perceptions of family affluence. Our analysis of the TIMSS 2019 
results confirmed these findings. The students from more affluent schools tended to 
achieve the best TIMSS mathematics scores in every system in the Dinaric region 
except Kosovo. In five participants, the mathematics achievement of students at more 
affluent schools was higher than that of students from more disadvantaged schools, 
with the biggest achievement gaps in North Macedonia (44 points) and Albania (39 
points). In Bosnia and Herzegovina and Kosovo, there was no statistically significant 
difference between these groups. 

As with mathematics, students from more affluent schools tended to achieve the 
best TIMSS science scores in every system in the Dinaric region except Kosovo. In six 
participants, the science achievement of students at more affluent schools was higher 
than that of students from more disadvantaged schools, with the biggest achievement 
gaps in North Macedonia (50 points) and Albania (42 points). In Kosovo, there was 
no Statistically significant differences between these groups. 

We also assessed results related to the TIMSS scale “Teaching Somewhat or Very 
Limited by Students not Ready for Instruction” (Mullis et al., 2020, exhibits 10.10 
and 10.11). Teachers generally reported that relatively few limitations were created 
by students who were not yet ready for instruction, at least in comparison with other 
TIMSS participants. In Albania, 71% of students attended schools that were affected 
“very little” by students not ready for instruction; in Kosovo 63% of students attended 
schools that were affected “very little” and, in North Macedonia, this was 60%. In 
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the other participants, less than half of the students had teachers who reported facing 
few issues (49% in Croatia and Serbia, 46% in Montenegro, and 45% in Bosnia and 
Herzegovina). At least a third of students in the region had teachers that reported 
experiencing “some” or “a lot” of limitations due to students not ready for instruction. 


3.3 School Climate: Safety and Order at Schools 


When we assessed perceptions of safety and order in schools, we found that teachers’ 
perceptions of this dimension of school climate differed quite considerably across 
the region (Table 3). 

Teachers of almost all students in Albania perceived their schools as very safe and 
orderly places, but only about half of the students in Croatia had teachers who thought 
their schools were very safe and orderly. In general, across the Dinaric region, only 
small percentages of students attended schools perceived by their teachers as “less 
then safe and orderly” (<3%), and, in most participating systems, except Croatia, 
there were also fairly low percentages of students in schools that teachers perceived 
as “somewhat safe and orderly” (Table 3). 

According to students, student bullying was present and relatively widespread in 
the Dinaric region. The percentages of students who reported frequent (monthly or 
weekly) bullying ranged from 15% of students in Albania to 32% of students in North 
Macedonia. Numerous national and international reports have reported findings on 
school violence in the Dinaric region. For example, when looking at adolescent 
experiences, the United Nations Children’s Fund (2017) reported that a quarter of 


Table 3 Percentage of students by how safe and orderly their school environment was, according 
to their teachers 


Education Teacher reported school was: 
system Very safe and orderly Somewhat safe and Less than safe and 

(% students) orderly (% students) orderly (% students) 
Albania 97 (1.3) 43 (1.3) | 0 (0.0) 
Bosnia & 80 (2.8) | 14 (1.7) O (0.0) 
Herzegovina 
Croatia 47 (3.2) |52 (3.2) |1 (0.7) 
Kosovo? 91 (2.3) 9 (2.4) |0 (0.0) 
Montenegro 85 (1.7) 14 (1.7 |0 (0.0) 
North 76 (3.4) | 22 (3.4) | 1 (0.7) 
Macedonia 
Serbia? 73 (4.2) | 24 (4.0 3 (1.4) 


Notes Standard errors appear in parentheses 
*National defined population covers 90-95% of the national target population 
Source Mullis et al. (2020), exhibits 8.7 and 8.8 
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students in Albania and North Macedonia experienced bullying in schools. Dinaric 
educational systems have strongly promoted zero violence policies in schools in 
response to this problem, and prevention programs have also been developed to 
tackle internet and cyber-bullying. 

We analyzed the TIMSS 2019 data on bullying at school level in relation to school 
material resources, for both mathematics and science, and identified no significant 
differences between the schools belonging to the groups with few and many resources 
for learning (Figs. 5 and 6). 

In general, we note that the education systems that scored higher on the Indexes of 
School Material Resources were not experiencing lower levels of bullying in schools. 
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Fig. 5 Percentages of students being bullied monthly or weekly in schools versus school resources 
for learning mathematics. Note In Kosovo and Serbia, the national defined population covers 90-95 % 
of the national target population 
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Fig.6 Percentages of students being bullied monthly or weekly in schools versus school resources 
for learning science. Note In Kosovo and Serbia, the national defined population covers 90-95% of 
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A focus on developing more intangible elements, such as a supportive school climate, 
a culture of achievement, and trust in school as an institution, may result in better 
environments for teaching and learning within schools. 


3.4 Impact of the Schools’ Material Resources, Environment, 
Composition and Climate on the Achievement Results 
(Regression Analysis) 


Having investigated the effects of several school-related factors on achievement, we 
undertook multivariate regression modeling to obtain a more comprehensive picture 
how all these factors were interrelated with achievement. The regression analyses 
revealed that the importance and significance of the factors varied across the region. 
We found that the Index of School Material Resources, and the school environment 
and climate factors explained only two percent of variance in student achievement 
in mathematics in Bosnia and Herzegovina, Croatia, and Montenegro, six percent of 
variance in Serbia, seven percent in Kosovo, and up to 11% of variance in Albania 
and North Macedonia (Table 4). The regression models also only explained two 
percent of variance in student achievement in science in Bosnia and Herzegovina, 
Croatia, and Montenegro, seven percent of variance in Kosovo and Serbia, and up 
to 12% of variance in Albania and North Macedonia (Table 5). The low power of 
variables related to school resources, school environment, and school climate in 
explaining student achievement strongly suggests that factors related to students’ 
home resources and the personal characteristics of students (interests, motivation, 
beliefs), and teachers’ and teaching characteristics together play a much greater part 
in supporting student achievement, as other chapters in this book confirm. 


4 Conclusions 


Around the world, education authorities are interested in supporting better learning 
for all and international large-scale assessments play a critical role in identifying and 
supporting solutions that affect student achievements (Mihaljevié Kosor et al. 2019). 
Although ILSA results sometimes lead researchers and policymakers to suggest that 
student achievement can be improved simply by something as obvious as investing 
in material resources, our research reveals that the answers are much more complex. 
Looking at the Dinaric region alone, factors related to material resources, school envi- 
ronment, and school climate did not show uniform or particularly strong effects on 
student achievement, although there were some interesting patterns that were aligned 
with the wider international results. In the TIMSS 2019 international results, higher 
average achievement in mathematics and science at grade four was associated with 
fewer school resource shortages and higher school emphasis on academic success 
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(Mullis et al., 2020). Regarding some elements of school climate, higher average 
achievement in mathematics and science, at both grade four and grade eight, was 
associated with students having a greater sense of school belonging and experiencing 
little or no bullying. At the system level, the results of PISA 2018 for 15-year-olds 
also indicated “that instruction hindered by a lack of educational materials was asso- 
ciated with lower reading scores in all participating countries and economies. School 
systems that showed more equity in the allocation of material resources tended to 
score higher in reading” (OECD, 2020, p. 196). 

We found that the amount of material resources in schools was related to grade 
four students’ mathematics and science achievement in four of the Dinaric partici- 
pants (Albania, North Macedonia, and Serbia), and related only to their mathematics 
achievement in Croatia. We found that schools with more students coming from 
affluent backgrounds tended to have the highest achievement in every participating 
system except Kosovo. Other research found that that there was a stronger emphasis 
on academic success in schools that are better equipped (see chapter “Characteris- 
tics of Principals and Schools in the Dinaric Region”). According to their teachers’ 
perceptions, almost all students in Albania to about half of the students in Croatia were 
taught in very safe and orderly schools. There was not a high prevalence of bullying 
in the Dinaric region, although around a third of students in Croatia and North Mace- 
donia reported that they were bullied monthly or weekly; this is a worrying level of 
bullying, and educational professionals in the region should devote more attention 
to finding solutions to tackle this issue. 

Although many education systems in the Dinaric region still have much to improve 
in terms of equipping schools with better material resources, our study highlights 
the importance of effective practice, and developing a supportive school climate 
and culture of achievement. “Ensuring that all schools have adequate and high- 
quality material resources, and the appropriate support, is key if students from all 
backgrounds are to be given equal opportunities to learn and succeed at school” 
(OECD, 2020, p. 16). As the definition of school material resources has broadened 
to include ICT skills and the associated digital tools and resources, school systems 
face a whole new level of procurement. 

Our study has confirmed that, beside the physical environment and material 
resources that support learning in schools, there are additional, less tangible dimen- 
sions of school life, which are equally important for the successful achievement 
of educational goals. The most important task of educational systems and school 
authorities is still to set and maintain both material factors (resources) and social 
factors of school functioning (such as safety, order, support, and emphasis on achieve- 
ment goals), and often the core aim is to improve student achievement. But, ideally, 
schools should provide equal opportunities for students that come from challenging 
or deprived environments; it is important that schools are not just buildings but 
also active catalysts of change through learning processes. Theory and ILSA results 
suggest that a good physical environment and sufficient material resources, together 
with supportive teachers, the existence of peer practices (for teachers and students), 
innovative methods, an open climate for discussion, and willingness to cooperate 
with parents and/or the wider community, establishes a productive setting for better 
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learning outcomes. Where schools do not have a shortage of material resources 
(such as space, equipment, or staffing), a critical factor for success is supporting 
healthy social relationships and fostering an open school climate, providing a school 
environment free from bullying and other stress factors. Our analyses showed that 
school-level variables only explained low levels of variance across the Dinaric 
region; consequently we conclude that home resources, the sociocultural capital of 
parents/guardians, and their willingness to participate in their child's schooling must 
play a major role in student achievement, together with students” attitudes toward 
the subject matter and their schools. While upgrading the material aspects of the 
educational environment is something that schools can influence and work hard on 
improving, good results can also be obtained by creating strong and healthy teaching 
and learning communities. 
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have dedicated considerable attention to teachers and their instructional practice in 
their efforts to improve student outcomes. The professional characteristics of teachers 
and their classroom behaviors may be important in determining how students acquire 
knowledge and develop skills in mathematics and science, and the relationships 
between teacher quality, instructional practice, and grade four student outcomes are 
consequently of great interest to researchers and policymakers. Analysis of IEA’s 
Trends in International Mathematics and Science Study (TIMSS) data indicates that 
grade four students are taught by teachers with similar educational backgrounds 
across the Dinaric region. Teacher quality (as measured by experience, level of educa- 
tion, and professional development) was related only to some aspects of instructional 
practice in the Dinaric region. Teacher quality was not a statistically significant 
predictor for student achievement in mathematics and science, although teachers’ 
formal education and years of experience were related to some aspects of student 
achievement. 
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1 Introduction 


Educators and researchers have consistently recognized and empirically shown that 
teachers and their classroom behaviors contribute more to student achievement than 
other systemic factors in education (Creemers & Kyriakides, 2008). Many countries 
have increased the educational requirements for class teachers in primary education 
to improve the quality of teaching and thereby student achievement in mathematics 
and science. Traditionally, formal education and experience are used as the principal 
measures of teacher quality (Burroughs & Chudgar, 2017). Formal teaching qualifi- 
cations may also include participation in continuous professional development (PD) 
(Nilsen et al., 2018). Goe (2007) defined teacher quality as a combination of teachers’ 
backgrounds (teacher qualifications and teacher characteristics), a process measure 
(teacher practices), and an outcome measure (teacher effectiveness). Some character- 
istics of teachers’ classroom behavior were ambiguous in terms of their relevance for 
student achievement in mathematics and science across education systems (Blómeke 
et al., 2016; Nilsen et al., 2018; OECD [Organisation for Economic Cooperation 
and Development], 2020); however, these studies also claimed that the professional 
knowledge and skills of teachers had equally important effects on student achieve- 
ment, regardless of the specific characteristics of education systems, teaching prac- 
tices, and student behavior in different settings. For example, cognitive activation, 
supportive classroom interactions, and classroom management have a positive effect 
on students’ achievement in mathematics and science (Decristan et al., 2016). 

In short, the importance of teacher characteristics, instructional practice, and their 
relation to student achievement is evident from the literature. In this chapter, we 
examine the status of the teaching profession, initial education, and professional 
development, describing the similarities and differences across the education systems 
in the Dinaric region. Our regional analyses of IEA's Trends in International Math- 
ematics and Science Study (TIMSS) 2019 provide in depth information about the 
relationships between the quality of teachers, instructional practice in participating 
classes, and grade four student outcomes on the TIMSS test. This supplies an evidence 
base for future investigation into the effectiveness of the strategies for improve- 
ment suggested by this research. Seven participants from the Dinaric region took 
part in TIMSS 2019, namely Albania, Bosnia and Herzegovina, Croatia, Kosovo, ! 
Montenegro, North Macedonia, and Serbia. 


2 Teachers and the Teaching Profession in the Dinaric 
Region 


Teachers who work in primary schools (e.g., at grades one to four) are called class 
teachers in all education systems in the Dinaric region. The classes are taught by 


! AU references to Kosovo in this document should be understood to be in the context of United 
Security Council resolution 1244 (1999). 
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one teacher, except in specific cases, when some subjects may be taught by special- 
ized subject teacher (e.g., English language). In the majority of TIMSS classes in 
the Dinaric region, class instruction is delivered in official languages and/or the 
languages of national minorities; the region is ethnically and culturally diverse. 
Most teachers of lower grades in primary school acquire their degree from teacher 
training faculties (state and/or private) across the Dinaric region. These faculties 
are oriented towards pedagogical, methodical, and didactical studies into subjects 
taught in primary education. In Albania, Croatia, Kosovo, and Serbia, a master’s 
level qualification (ISCED [International Standard Classification of Education] 7; 
see UNESCO [United Nations Educational, Scientific and Cultural Organization] 
Institute of Statistics, 2012 for an explanation of the ISCED classifications) is the 
minimum level of initial teacher education required for employment in primary 
schools. In Bosnia and Herzegovina, Montenegro, and North Macedonia, a bach- 
elor level qualification (ISCED 6) is required for teachers of grade four students. 
After completing their academic studies, to gain employment as a teacher, candidate 
teachers must also pass a state examination for teacher certification (except in Bosnia 
and Herzegovina). In most education systems in the Dinaric region, young teachers 
enter an initiation program at the beginning of their careers to introduce them to the 
teaching profession. Professional orientation is provided through different types of 
mentoring by more experienced teachers. All education systems in the region have 
an induction period, which varies slightly in duration and ranges from a minimum 
of six months to, more commonly, a full year of probation before a teacher can be 
fully registered as a professional (Pantić et al., 2011). 

Teachers from the Dinaric region have an obligation to develop professionally 
during their careers by attending state-organized training courses. The state agencies 
and institutes have the leading role in creating professional development policies 
and in their implementation. While some education systems in the Dinaric region 
have mechanisms in place to accredit providers and programs, others are struggling 
to implement a coherent system (e.g., Bosnia and Herzegovina). Across the region, 
training programs are provided by public, scientific, and professional associations 
and/or private institutions. Most teachers from the region choose the programs they 
wish to attend from a list of accredited training programs approved by the state 
agencies. 

There are between-systems differences in the amount of time teachers need to 
dedicate to PD across the Dinaric region. Croatian teachers are obliged to partici- 
pate in PD programs in accordance with a proscribed number of hours at the state, 
county, and school level (Elezović & Muraja, 2020; Viorel, 2017). Teachers from 
Kosovo, depending on career development paths, and on the criteria and conditions 
for licensing by the type of license, must ensure that they have the number of required 
hours of PD (Mehmeti et al., 2019). Teachers in North Macedonia are expected to 
log at least 60 h of PD over three years (OECD, 20192), while teachers from Albania 
must undertake at least three days training per year (Vrapi & Alia, 2020). Serbian 
teachers are required to undertake 64 h of various PD activities annually. A required 
number of teacher PD training hours is not stipulated at the state level in Bosnia and 
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Herzegovina or Montenegro (Duda et al., 2013; Popić & Džumhur, 2020). Never- 
theless, policymakers and school leaders need to ensure that PD opportunities are 
available for all teachers across the region. 

High-quality PD activities are recognized as crucial if education systems are to 
ensure that all teachers possess and maintain the relevant competencies to be effec- 
tive in modern classrooms (Viorel, 2017). Several studies have shown that teachers in 
the Dinaric region face very similar challenges concerning PD. Firstly, PD programs 
in the Dinaric area are usually designed as one-off seminars and courses (Pantić 
et al., 2011). In Bosna and Herzegovina, Kosovo, Montenegro, and North Mace- 
donia, teachers and researchers have expressed concern both about the quality and the 
availability of training courses (Mehmeti et al., 2019; Mićanović & Vučković, 2014; 
OECD, 20192). In Serbia, seminars are still often fragmentary, unrelated to teaching 
practice, insufficiently intensive, and lack the necessary follow-up and support, 
according to educational experts, school principals, school counselors, and experts 
in pedagogy and psychology (Đerić et al., 2014). Likewise, educational authorities 
have observed that teacher training in Bosnia and Herzegovina is outdated, does not 
follow current education trends, and does not support the progress of teachers (Popié 
& Džumhur, 2020). By contrast, the OECD’s Teaching and Learning International 
Survey (TALIS) 2018 found that a very high percentage of Croatian teachers (86%) 
reported that the PD activities that they attended had a positive impact on their 
teaching practice (OECD, 2019b). However, for most teachers across the Dinaric 
region, the budget devoted to PD is insufficient to access opportunities to become 
involved in PD activities (OECD, 2009; Viorel, 2017). For teachers across the region, 
which teacher competencies should be developed, which professional knowledge 
should be offered to teachers within PD training, how much time needs to be dedicated 
to PD, and how PD activities can be organized efficiently remain open questions. 

Teachers who participated in OECD's Programme for International Student 
Assessment (PISA) 2018 from the Dinaric region education systems "rely heavily on 
traditional pedagogy, such as lecturing to students and encouraging them to memo- 
rize information set out in the curriculum" (OECD, 2020, p. 65). This last report 
showed that pedagogical methods in the region (as perceived by students) were still 
largely traditional and associated with lower student performance. Prior to TIMSS 
2019, little data had been gathered on the quality of instructional practice in the 
primary grades of elementary school across the Dinaric region. In addition, there 
was a lack of comprehensive and joint research on the quality of instructional prac- 
tices in the Dinaric region, especially for mathematics and science in lower grades. 
Our analyses of the TIMSS 2019 data examines the relationship between quality of 
teachers, instructional practice in classrooms, and grade four students' achievement. 


3 Methods and Research Questions 


Over time, an extensive literature has been developed on teacher quality, instructional 
quality, and students’ outcomes based on international data (e.g., Đerić et al., 2017; 
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Nilsen & Gustafsson, 2016). Several education systems from the Dinaric region 
did not participate in previous TIMSS assessments of students in mathematics and 
science at grade four (e.g., Bosnia and Herzegovina, Kosovo, Montenegro, and North 
Macedonia). Added to this, there are not many regional or national studies exam- 
ining the relations between teacher quality, instructional practice, and outcomes for 
representative samples of grade four students in the Dinaric region. Our interest is in 
describing the “teachers’ profile” for the sampled TIMSS 2019 grade four classrooms 
in the Dinaric region and examining whether, and to what extent, teacher quality and 
instructional practice contribute to grade four student achievement in mathematics 
and science. We look in depth at: (1) teacher quality; (2) instructional practice; (3) the 
relationship between teacher quality and instructional practice; and (4) instructional 
practice as a factor related to student achievement in mathematics and science. 
For our analyses, we focused on three research questions: 


(1) What are the similarities and differences, in terms of teacher quality and 
instructional practices, across the different education systems of the Dinaric 
region? 

(2) Is teacher quality related to aspects of instructional practice across the Dinaric 
region? 

(3) Does the instructional practice of teachers contribute to student achievement 
when controlling for teacher quality? If so, does student achievement in math- 
ematics and science depend on the relationship between teacher quality and 
instructional practice? 


3.1 Sample and Data Sources 


Teachers who complete TIMSS questionnaires represent the teachers of a national 
sample of students (Martin et al., 2020). We used the data obtained from teacher 
questionnaires in conjunction with achievement test data measuring students' math- 
ematics and science outcomes. Instruments were administered in both the official 
language and minority languages of the respective education systems in the Dinaric 
region (except in Serbia, where materials were administrated only in the official 
language). Teachers' data are interpreted by the percentage of students who are 
taught by teachers with a specific characteristic. More general information about the 
analysis methods, sample characteristics, and data sources that we used are available 
in Sects. 5 and 5.1. 


3.2 Variables and Measures 


We identified several variables and scales in the TIMSS 2019 international reports 
as being of potential interest for our research (Table 1, see also Table S.11 in the 
supplementary materials at www.iea.nl/publications/RfEVoll13). 


156 


Table 1 List of the variables and scales used in analyses 


I. Derié et al. 


Variables 


Teacher’s years of 
experience 


Description 


Years of experience as a 
teacher altogether 


Values/Response options 


Number (years) 


References 


Fishbein et al. (2021, 
Supplement 3, p. 71) 


Teacher education 


Mathematics/science 
teachers’ major subject of 
study 


Teacher’s highest level of 
formal education completed 


Combination of teachers’ 
reports on major area of 
study and specialization 


Recoded from seven to three 

categories 

(1) Did not complete 
Bachelor’s or equivalent 
level 

(2) Bachelor’s or equivalent 
level 

(3) Completed postgraduate 
degree 


There were three categories: 

(1) Major in primary 
education and 
mathematics 

(2) Major in primary 
education but not in 


Fishbein et al. (2021, 
Supplement 3, p. 71) 


Fishbein et al. (2021, 
Supplement 3, p. 71) 


mathematics 
(3) Others 
Professional development | Numbers of hours teachers | (1) 16 h and more Fishbein et al. (2021, 
teaching devoted to professional (2) 6-15h Supplement 3, pp. 80 and 
mathematics/science development in teaching (3) Less than 6h 86) 


mathematics/science in the 
past two years (recoded 
from five to three 
categories) 


Professional development 
needs in teaching 
mathematics/science 


Teachers indication of 
needs for professional 
development in teaching 
mathematics (seven areas) 
or teaching science (eight 
areas) 


There were two response 
options per item: 

(1) Yes 

(2) No 


Fishbein et al. (2021, 
Supplement 3, pp. 80 and 
86) 


Instructional time for 
teaching 
mathematics/science 


Teacher reports on the time 
spent on teaching 
mathematics/science to the 
TIMSS class in a typical 
week 


Number (minutes) 


Fishbein et al. (2021, 
Supplement 3, pp. 76 and 
81) 


Instructional practice in 


Teachers reports on 


Index with four categories: 


Fishbein et al. (2021, 


mathematics frequency of asking (1) Every or almost every Supplement 3, p. 76) 

students to apply what they lesson 

have learned to new (2) About half of the 

problem situations on their lessons 

own every or almost every | (3) Some lessons 

lesson (4) Never 

(continued) 

Teacher Quality 


A set of questions in the TIMSS grade four teacher questionnaire (TIMSS & PIRLS 
International Study Center, 2018) asked teachers about their educational background: 
namely, their formal education, specialization, experience, the number of hours they 
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Table 1 (continued) 


Variables Description Values/Response options References 
Instructional practice in Teachers reports on Index with four categories: | Fishbein et al. (2021, 
science frequency of asking (1) Every or almost every Supplement 3, p. 81) 

students to use evidence lesson 

from experiments or (2) About half of the 

investigations to support lessons 

conclusions (3) Some lessons 

(4) Never 


had devoted to PD in teaching mathematics and science, and whether they had 
participated in PD during the last two years. 


Instructional Practice 


Teachers were asked to report how often they performed various activities in 
the TIMSS sampled mathematics and science classes they were teaching to (“In 
teaching mathematics/science to this class, how often do you ask students to do 
the following?"). Our measure of instructional practice is based on their responses 
to two of the items: namely, how often they asked students in their teaching to 
“apply what they have learned to new problem situations on their own” during math- 
ematics lessons, and “use evidence from experiments or investigations to support 
conclusions” during science lessons. 


Student Outcomes 


Student outcomes include both mathematical achievement and science achievement 
in the TIMSS 2019 test. Mathematical and science achievement are represented by 
five plausible values representing student achievement and all five plausible values 
were used in our analyses. 


4 Results and Discussion 


4.1 Teacher Quality in the Dinaric Region 


The educational background of mathematics and science teachers was similar across 
the Dinaric region. Most grade four students had teachers that possessed a bachelor's 
degree or an equivalent qualification (ISCED level 6), but not a postgraduate degree. 
Teachers from Albania and Croatia had the highest levels of education; more than 
half of them had some kind of postgraduate university degree (M.A., Ph.D., or other 
postgraduate qualification). For most teachers from the Dinaric region, this level of 
formal education is in line with the policy recommendations and requirements of 
the European Union (EU) (Table 2). Several large-scale studies suggest that, while 
teachers in many education systems are well educated (Mullis et al., 2020; Schleicher, 
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Fig. 1 Percentage of students taught by teachers whose major subject of study was mathematics. 
Note In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population 


2020), there is still a notable percentage of teachers around the world that possess 
low levels of formal education. 

Most students in the Dinaric region had teachers who, on average, were slightly 
more experienced (Table 2) than their colleagues in the other education systems that 
participated in TIMSS 2019. Teachers from Bosnia and Herzegovina, Kosovo, and 
North Macedonia, on average, had less than 20 year of teaching experience, which 
was closer to the international average (mean — 17 years; Mullis et al., 2020). Serbian 
grade four students had most experienced teachers in the region. 

Teachers were also asked what their major or main area(s) of study were during 
their academic education. For most teachers, the focus of their academic educa- 
tion was teaching primary education, without any specialization in mathematics and 
science (Figs. 1 and 2); this finding is consistent with the pedagogical orientation 
of teacher training faculties across the Dinaric region. Most students had teachers 
whose major subject of study was teaching in elementary schools/primary education. 
Only very few students (<10%) were taught by teachers who studied for a degree in 
mathematics or science or another academic subject. The PISA 2018 report found 
no relationship between teacher qualifications and student outcomes in the Western 
Balkans (OECD, 2020). While a recent study showed that teacher specialization 
could be linked to effective teaching practices and student achievement of grade four 
students in Sweden (Johansson & Myberg, 2019), initial teacher education is often 
insufficient to prepare primary and secondary teachers for their challenging jobs. 
Highly qualified teachers must possess full state certification, a master's degree, and 
demonstrate subject matter competency in each of the academic subjects they teach. 
Teachers also need high quality PD activities to develop relevant competencies to be 
effective in modern classroom conditions (OECD, 2020; Viorel, 2017). 
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Teachers' major: 


^ Major in primary education 
and science 


= Major in primary education 
but not in science 


Others 


Percentage of students (%) 


Fig. 2 Percentage of students taught by teachers whose major subject of study was a science. 
Note In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population 


As part of the TIMSS 2019 teacher questionnaire, teachers in the Dinaric region 
reported how many hours they had spent undertaking formal PD activities (such as 
workshops and seminars) related to teaching mathematics and science over the last 
two years. We divided the data collected into the following three categories: (1) 16 h 
and more; (2) 6-15 h; and (3) less than six hours. In general, the level of PD across 
the region was low (Figs. 3 and 4). 

Overall, Dinaric grade four students were taught by class teachers that spent more 
time on PD activities that were related to teaching mathematics than teaching science, 
although the difference was not prominent (except for Albania and Montenegro). 
Almost 4046 of the students from Kosovo had teachers who devoted 16 h or more 
PD to teaching mathematics, which is significantly higher level than was reported 
by teachers from other education systems in the Dinaric region. Alarmingly, more 
than half the students from Bosnia and Herzegovina (85%), Croatia (65%), North 
Macedonia (61%), and Albania (5496) were taught by teachers who reported dedi- 
cating less than six hours to PD in mathematics over the previous two years. The 
figures reported for teacher PD related to teaching science were even lower than 
those for mathematics. A large percentage of grade four students from Bosnia and 
Herzegovina (8196) and North Macedonia (7696) have teachers who dedicated less 
than six hours to PD in science. The largest percentage of grade four students (2396) 
whose teachers reported spending 16 h or more on PD in science was in Kosovo. 

Teachers of the grade four students in the Dinaric region spent more time on PD 
for mathematics than for science teaching, but there was also wide variation among 
teachers across the region in terms of overall time invested in PD. Teachers from 
Kosovo reported investing the greatest amount of time on PD in mathematics and 
science, while teachers from Bosnia and Herzegovina, Croatia, and North Macedonia 
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Fig. 3 Percentage of students taught versus number of hours their teachers devoted to professional 
development in teaching mathematics over the previous two years. Note In Kosovo and Serbia, the 
national defined population covers 90-95% of the national target population 
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Fig.4 Percentage of students taught versus number of hours their teachers devoted to professional 
development in teaching science over the previous two years. Note In Kosovo and Serbia, the 
national defined population covers 90-95% of the national target population 
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reported little time was invested on PD. Recent study shows that long-term PD 
programs are more effective, both in terms of the overall amount of time that the 
activity takes, and the total amount of hours spent (Barrera-Pedemonte, 2016). 

Teachers were further asked if they had participated in content-specific (but not 
necessarily formal) PD activities over the previous two years. The question included 
the following categories of answers for both subject areas: (1) content; (2) peda- 
gogy/instruction; (3) curriculum; (4) integrating technology into instructions: (5) 
improving students’ critical thinking or problem-solving skills; (6) assessments; (7) 
addressing individual students’ needs; and (8) addressing students’ language needs 
in learning mathematics or science. However, grade four teachers across the whole 
region stated that the most pressing future need in the field of mathematics and 
science PD was integrating technology into instruction (Tables 3 and 4). This is in 
line with the development and application of technologies to other areas of society, 
and growing interest in teaching children and youth how to use ICT at school and in 
everyday life (IEA, 2021). 

Many teachers across the region noted a need for PD in addressing individual 
student needs and improving students’ critical thinking and problem-solving skills 
(Tables 3 and 4). Their interest in these themes indicates that teachers of grade four 
students are aware of the generic competencies they should focus on developing in 
their students. We can postulate that a desire for support to help them develop skills 
in innovative teaching methods is seen as a way to improve teaching efficiency and 
enhance students’ results. Analysis of the TIMSS data across many education systems 
supports the conclusion that students of teachers at grade four who improved their 
professional knowledge of mathematics content through undertaking PD activities 
tend to have higher achievement scores than other students (Liang et al., 2015). 


4.2 Instructional Practice in the Dinaric Region 


There was large reported variation in the time devoted to mathematics and science 
instruction among the education systems in the Dinaric region (see also chapter 
“Opportunity to Learn Mathematics and Science”). On average, grade four students 
from Kosovo and Serbia received significantly more hours of mathematics teaching 
per week than other students in the region (Fig. 5). The time spent on science lessons 
showed even greater variation, ranging from an average time of 92 min per week in 
Albania to 137 min per week in Croatia. 

According to the TIMSS 2019 data, the amount of instructional time that students 
spent in classrooms per week varied widely by subject in the Dinaric region (Fig. 5). 
In four of the education systems (Albania, Kosovo, Montenegro, and Serbia) grade 
four students spent at least twice as much time on mathematics compared to science. 
Across the region, students from Serbia (245 min per week) and Kosovo (240 min 
per week) spent the greatest amount of time on learning mathematics, while students 
from Croatia devoted the greatest amount of time to learning science (137 min per 
week). Recent research has shown that the amount of time that students spend on 


163 


uone[ndod 4931¥} [euoneu au Jo 9566-06 S129409 uone[ndod pouyop IEuOnEN, 
sosoyjuosed ur reodde sio.ə prepuvjg sajoyy 


Egal 6v | (Qt) 9€ | (ct) 9€ | (0) v9 | (S'£) LT | (çO) O€ | (€e) vC lU 

pIuopoov] 

(8'2) SS (rr) Tr | rr) vs (cv) €9 | (8'7) Er | rt) Er | (rp cr VON 

(LO) €9 | (8'7) OS | (çO 99 (pO I8 | (87) LS | (TO 6v GO OS | o1douojuojq 

Ier) 08 | (9€) €8 | (Oe) €8 | (o£) v8 IOC) v8 | (O'S) pL | (€S) vL pOAOSOY 

(LO) E8 | (C'E) LL | (97) L8 | (O 06 | (S€) 8S | (9'8) 6S | (ee) €9 POH 

PUIAOS9ZIH 

Egal vs | (re 9 | (p £9 | (S£) TL | (87) 8c | (0.9 o£ | (ro 6  3enusog 

(cv) 6S | (Ç?) TS | (cv) 09 | (ee) 8L | (6€) LS | ÇO €9| (e) ES verueqry 
spoou s,juopnis site Suyu) uononJjsur 
TenprArmpur Teonuo ,sjuopnijs ur ÁSo|ouuo91) 

Suisso1ppy juouissossy SurAo1dui[ Zum at umnəruno uononnsul/ÁSosSepəoq JuəoluooO wayshks 

0} Daigtat Juəutdo[ƏAəp [euorssəJoid 104 poou v pojeorpur s1oqoeo) ƏsouA (9%) sluəpmis Jo e3vjuooroq uoneonpq 


Teachers, Teaching and Student Achievement 


soneuroqjeui JUYI) 0} PAPJA juourdo[oAop [euorssojo1d ərmnj 10g spoou SNOLIVA pojeorpur SIIYILIA ƏsouA sjuopnis Jo sosvjusdIog € ABL 


I. Đerić et al. 


uone[ndod jo31e) peuoneu ou) Jo %ç6—06 S19Ao5o uone[ndod pouysp [euoneN., 
sosoujuored ur 1eodde s10110 prepuejg sajoyy 


164 


(Ep) vS | ECO br | (Çe) 9c | Grp) vs | (60 19 | (89) ce | OH) 8€ | (S'E) 0€ wies 

eIUOpooeJA 

(v) ES | (Ç) TS | (t) 8r | (€p) v9 | (Ç?) 09 | (r$) £r | (€ Lr | (Ç?) Sr WON 

(ro (8'7) 9c | (Sg 6S | o13ouojuojA 

(ce (tp) LL | (ro 08 yOAOSOY 

(ro (çO) 19 | (9'7) IL POOL) 

BUIAOSOZIOH 

(6€) (o's) ve | (ED) Fr 39 usog 

(Tt) (Tp 09 | (T) Sc MAI 
sjooíqns 1oo spoou s,juopnis Sys Suyup uononnsur 
(DIA oouoros Tenpramur Ieonuo ,sjuopnis ur AÁSo[ouuo9) 

SunviSoju[ Surssoppy juouissossy SurAo1duiT Sunessoquy umnərumo uonənnsul/K8osepəq Juəjuoo uosis 

0} poje[o1 juouidojoAop [euorssojoJd 10} poou e pojeorpur SIayovay osouA (%) sJuəpnis Jo oSvjuoo1oq uoneonpq 


oouoros JUYI 0} pIJ juouido[oAop [euorssogoJd omnj 10] spoou snormA pojeorpur srəuoeə) osoqA sjuopnjs Jo sogvjuoolod p o[qer 


Teachers, Teaching and Student Achievement 165 


Albania 190 Average instructional 
92 time for: 


. ; 198 m Mathematics 
Bosnia & Herzegovina 121 
m Science 
i 204 
Croatia 137 
240 
Kosovo Eu 
195 
Montenegro 
3 124 
North Macedonia 217 
o cedoni B wm 
" 245 
Serbia 128 


Instructional time (minutes per week) 


Fig. 5 Average instructional time spent on teaching mathematics and science per week (minutes) 
Note In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population 


learning mathematics and science on weekly basis in Eastern European educa- 
tion systems is significantly related to student achievement (Lavy, 2015). Some 
authors have suggested that, based on extended analysis of international data, “dif- 
ferences in instruction time play a less important role than previously thought for 
explaining international gaps in student achievement" (Bietenbeck & Collins, 2020, 
p. 9); however, this divergence among international studies may be partly due to the 
differing criteria used to measure instruction time in the international data. 
Teachers in different education systems have different teaching styles, shaped 
by beliefs and attitudes about teaching, and what they have learned during their 
initial teacher preparation programs and during subsequent PD. To better understand 
those different teaching styles, and investigate which styles were more successful, the 
TIMSS 2019 teacher questionnaire asked about specific activities that teachers under- 
took during their mathematics and science lessons (Tables S.12 and S.13 provide 
more detailed results for both mathematics and science, respectively; see supple- 
mentary materials at www.iea.nl/publications/RfEVol13). A very high percentage of 
grade four students (28096) had teachers who stated that in almost half of the math- 
ematics lessons, students listened while teachers explained new content in math- 
ematics or demonstrated new ways of problem solving or just memorizing rules, 
procedures, and facts. More than 90% of students across the whole region were asked 
by their teachers in at least half of their mathematics lessons to practice procedures 
on their own and apply what they have learned to new problem situations, except in 
Albania and Kosovo, where this percentage was drastically lower. In all participating 
education systems, working in mixed ability groups was more common than working 
in groups with similar abilities for both mathematics and science lessons. In science 
classes, the most common instruction activities that grade four teachers from the 
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region applied were explaining new content to the students, reading textbooks and 
other sources, and memorizing facts and principles. Observing and describing natural 
phenomena, such as weather or plant growth, was also frequently done. Teachers were 
less likely to ask students to work more independently and creatively, or to work on 
activities requiring higher order cognitive skills. Examples of such activities include 
designing and conducting experiments, presenting, and interpreting results and using 
them to draw conclusions. It is noticeable that, according to the teachers' reports, 
students in Albania and North Macedonia engaged in such activities more often than 
other students in the Dinaric region. 

In teaching mathematics and science in the Dinaric region, the instructional 
methods of problem-solving, research, and experimental teaching methods were 
not sufficiently represented. Our results can be compared with a previous anal- 
ysis of TIMSS 2015 data for teaching practices in Serbia, Croatia, and Hungary 
(Derié et al., 2017). TIMSS 2019 results showed that most students of the teachers in 
the region implemented procedures that were more teacher-centered, while students 
played largely passive roles during mathematics and science classes. For example, 
the data indicate that students of science listened to teachers explain concepts, read 
lessons from the textbooks, and they remembered the facts and principles (Mullis 
et al., 2020). These instructional practices are very important when building the basic 
knowledge of younger students, especially in the fields of mathematics and science. 
Nevertheless, it was relatively rare for teachers to use innovative teaching practices, 
such as asking their students to plan and conduct experiments or work in the field 
and outside the classroom; this reinforces earlier TIMSS findings in Serbia (Mirkov 
& Lalić Vučetić, 2018), as well as in other education systems in the region (Martin 
et al., 2016; Mullis et al., 2016, 2020). 

PISA 2018 also found that teachers in this region were using less adaptive instruc- 
tion and more teacher-directed instruction (OECD, 2020). Teachers may choose more 
traditional roles and procedures, believing that these are effective ways of working 
with grade four students, or they may lack the confidence (either in themselves or their 
students) to apply more innovative methods. But, with appropriate support, students 
of this age can be effectively engaged in investigation, gathering and analyzing data, 
and in drawing conclusions based on evidence (Derié et al., 2017, 2020; Mullis et al., 
2020). 

Students participating in TIMSS 2019 reported that they know what their teachers 
expect from them, that the teachers explain contents clearly, and that teachers answer 
their questions and provide help and support in learning. Compared to their peers 
across the region, students from Croatia and Serbia were less likely to agree that 
their teachers applied these instructional practices in mathematics and science classes 
(Mullis et al., 2020). It seems that students in the Dinaric region generally perceive 
traditional forms of teaching and learning as engaging. Fauth et al. (2014) stressed 
that it was necessary to be cautious when interpreting such data because, with students 
of this age, the overall popularity of the teacher affects the student's evaluation of 
the quality of their classes. 
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4.3 Relationship Between Teacher Quality and Instructional 
Practice in the Dinaric Region 


Recent evidence from international studies has suggested that teacher quality is 
significantly related to instructional quality (Blómeke et al., 2016). To establish 
whether teacher quality was related to instructional practice across the Dinaric region, 
we investigated the quality of teachers as a construct expressed by the length of their 
teaching experience, level of formal education, and time dedicated to PD (more than 
15 hours), and examined the relationship of this construct with instructional practice 
of teachers for both mathematics and science. However, we found that there was 
no consistent relationship between teacher quality and instructional quality across 
the Dinaric region; teacher quality indicators were related to only a few aspects of 
instructional practice or not at all. In some cases, teachers who were more experienced 
and better educated, and those who spent more time on PD activities, appeared to be 
more willing to use cognitive-activation strategies that require students to use higher 
levels of thinking (e.g., use evidence from experiments or investigations to support 
conclusions). 


4.4 Instructional Practice as a Factor in Student 
Achievement in the Dinaric Region 


To investigate whether the characteristics of teachers and classes can be used as 
predictors of student achievement in mathematics and science, we undertook multi- 
variate linear regression analyses (see Chapter 1). Such multilinear modeling aims to 
answer whether the instructional practice of teachers is related to student achievement 
when teacher quality variables are controlled, and vice versa. Both mathematics and 
science models explained less than three percent of the variance in student achieve- 
ment in mathematics and science; few predictors were significant, and their contri- 
butions were small (Tables 5 and 6). Thus, even if the factors related to professional 
characteristics of teachers and the quality of their teaching had shown to be signif- 
icant predictors of achievement, they would only have explained a small amount of 
the achievement in mathematics and science. 

Across the Dinaric region, we found that teacher quality measures were not statis- 
tically significantly predictors of student achievement in mathematics and science, 
although there were some exceptions where their level of formal education and years 
of the working experience had an effect (see Sect. 4.3). Our findings are consistent 
with other studies that noted “measurable” teacher characteristics explained only a 
small portion of the variance in student achievement (Derié et al. 2017; Munoz and 
Chang 2007), and this creates a clear dilemma for policymakers. In general, this lack 
of variance in developed education systems contributes to the problems associated 
with observing an impact on learning outcomes. So, instead of focusing on identi- 
fying differences among teachers who have increasingly similar backgrounds, it is 
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equally important to improve the processes involved in the preparation, recruitment, 
mentoring, promotion, and dismissal of teachers (Rivkin et al. 2005). 


5 Conclusions 


There were previously few national and/or regional studies examining the relation- 
ships between teacher quality, instructional practice, and student outcomes in the 
Dinaric region. Here, we were able to use data based on representative samples of 
grade four students from TIMSS 2019, together with TIMSS measures of teacher 
quality and instructional practice, to analyze the contribution of these teacher 
variables to student achievement in mathematics and science across the Dinaric 
region. 

The educational background of mathematics and science teachers is similar across 
the Dinaric region. Teacher education in the region increasingly follows existing EU 
requirements. Most students in the Dinaric region have teachers who, on average, 
are slightly more experienced than their colleagues in other education systems that 
participated in TIMSS 2019, however, the level of PD for those teaching mathematics 
and sciences is quite low across the Dinaric region. Responses to the TIMSS 2019 
teacher questionnaire indicate that mathematics and science teachers in the region 
are aware of which competencies they should develop in students and that they need 
support in acquiring more innovative teaching methods. The future needs for PD 
that they identified are consistent with current trends in the field of education and 
new social circumstances regarding the use of ICT. Decision makers should take 
these teacher observations into account and adjust future PD activities accordingly. 
Facilitating easier access to PD opportunities and raising the quality and relevance of 
these programs can also increase teacher participation and help teachers to strengthen 
their practice, knowledge, and skills (OECD, 2020). Policymakers and teachers in the 
Dinaric region could use this information to improve PD and control the successful 
implementation of changes in the next TIMSS cycle. 

According to our analyses, teacher quality measures were not statistically signif- 
icant predictors for student achievement in mathematics and science in most educa- 
tion systems in the Dinaric region. Teacher quality was related only to some 
aspects of instructional practice. TIMSS 2019 data (Mullis et al., 2020) showed 
that most teachers in the Dinaric region based their standard practice on more tradi- 
tional teacher-centered activities (e.g., students read lessons from the textbooks and 
remember the facts and principles), while modern teaching methods suggest that it is 
beneficial for students to play more active roles in mathematics and science classes. 
Dinaric teachers who are more educated, more experienced, and those who spend 
more time on PD activities are more willing to use cognitive-activation strategies that 
require students to use higher cognitive levels of thinking (e.g., use evidence from 
experiments or investigations to support conclusions). 
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Even though teacher quality and instructional practice have not been shown as key 
factors in predicting student achievement in mathematics and science, their impor- 
tance should not be overlooked. It is necessary to be cautious when interpreting the 
results and to carefully review the different aspects. Precisely which characteristics 
and behaviors of teachers in the classroom affect student achievement in the Dinaric 
region remains unclear. Effectiveness studies conducted over several decades on 
diverse hierarchical levels (individual, class, and school level) provide some answers 
(Creemers & Kyriakides, 2008), giving us an opportunity to get closer to describing 
the ideal profile of an efficient teacher, who can optimally guide and support their 
students. Such studies provide information on possible identification and system- 
atization of student, teacher, and school characteristics that influence achievement, 
enabling improvement in teaching practices and overall quality (Teodorovic, 2011). 
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Abstract The principal in a school is a manager and school leader who is respon- 
sible for advancement and implementation of various processes in the school. They 
take responsibility for compliance and accountability, support the teaching staff in 
their professional activities, and aim to build and maintain a school environment that 
promotes student achievement and good school-community relations. An important 
element of IEA’s Trends in International Mathematics and Science Study (TIMSS) is 
the research into the home, community, school, and student factors associated with 
student achievement in mathematics and science. To accomplish this, data about the 
contexts for learning are collected through questionnaires completed by students 
and their parents/guardians, teachers, and school principals. Analysis of data from 
TIMSS 2019 for participants in the Dinaric region (Albania, Bosnia and Herzegovina, 
Croatia, Kosovo, Montenegro, North Macedonia, and Serbia) was used to determine 
whether the level of education, years of experience of the principal, the location of the 
school, and/or school composition have significant effects on student achievement 
and thus potentially identify elements that facilitate academic achievement among 
students. The TIMSS school questionnaire asks principals to provide assessments 
of the literacy and numeracy skills of students when they first start schooling, the 
socioeconomic background of the students attending the school, the availability of 
instructional resources, the school’s emphasis on academic success, and discipline 
and school safety. While previous research has suggested that principals’ years of 
experience and educational attainment are positively related to student achievement, 
there was little evidence for this in this regional sample of education systems. In three 
of the seven TIMSS participants in the Dinaric region, students from schools with a 
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socioeconomically more affluent student body tended to achieve higher scores in 
mathematics and science. In four of the seven TIMSS participants, schools that placed 
strong emphasis on academic success tended to have higher levels of student achieve- 
ment. The findings suggest that school principals in the region can best improve their 
students’ achievement by focusing on encouraging student motivation and providing 
additional instructional resources for socioeconomically disadvantaged children. 


Keywords Achievement * Leadership - Mathematics * School education * School 
principals * Science - Trends in International Mathematics and Science Study 
(TIMSS) 


1 Introduction 


What organizational features make a school a better place for teachers to teach and 
for students to learn has always been a very important question. Hoy (2012) identified 
three characteristics of schools that made a positive difference to student achievement 
after controlling for socioeconomic status (SES), namely (1) collective efficacy, (2) 
collective trust in parents and students, and (3) the academic emphasis of the school. 
In addition, school location can play an important role in education; schools in urban 
areas differ from schools in rural areas, and usually the former are associated with 
higher student performance. There are several explanations for this. Urban schools 
are usually larger, enjoy greater responsibility for resource allocation, and are less 
likely to experience staff shortages. Urban schools tend to have a higher proportion of 
qualified teachers, and higher student to teacher ratios than schools in rural areas and 
towns (OECD [Organisation for Economic Cooperation and Development], 2013; for 
additional research into the effects of school location, see also chapter “Scaffolding 
the Learning in Rural and Urban Schools: Similarities and Differences"). 

In public schools (state-funded), principals are also responsible for implementing 
standards, programs, and regulations set by higher educational authorities (such as 
government ministries) and related bodies. The role that principals play in schools 
is very important, as they are not only the administrators but also the initiators of 
many processes. As school managers, they should be school leaders who improve 
school processes and support high achievement among students (Malere & Ozola, 
2019). Principals serve as the public representatives of their school. Elementary 
school principals provide direction and manage the overall operations of schools. 
They set and oversee academic goals, and ensure that teachers have the equipment 
and resources to meet those goals. Principals may establish and supervise additional 
programs in their school, such as counseling, extracurricular activities, and before- 
and after-school daycare. Principals clearly have an important management role, 
including responsibilities for teachers, curricula, and school budgets. They further 
facilitate cooperation with the students’ parents and the local community by listening 
to and addressing their concerns. Research has indicated that school environment 
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created by the principal may have a significant influence on students’ mathematics 
and reading achievement (Alhosani et al., 2017; Dhuey & Smith, 2014). 

Seven participants from the Dinaric region took part in TIMSS 2019, namely 
Albania, Bosnia and Herzegovina, Croatia, Kosovo,! Montenegro, North Mace- 
donia, and Serbia. Across the Dinaric region, elementary school principals typically 
undertake the following activities: 


manage school activities and staff, including teachers and support personnel; 
establish and oversee class schedules; 

implement and maintain curriculum standards; 

observe and evaluate teachers’ performance; 

meet with parents and teachers to discuss students’ progress and behavior; 
assess and prepare reports on test scores and other student achievement data; 
organize professional development programs and workshops for staff; 

establish and coordinate security procedures for students, staff, and visitors; and 
manage the school budget, including the provision of school supplies, and 
maintenance. 


Some research has suggested that the “formal” characteristics of principals may 
play an important role in student attainment. For example, a systematic review by 
Osborne-Lampkin et al. (2015) reported that principals’ years of experience and 
educational attainment were positively related to student achievement. 


1.1 Framing the Research Questions 


Our research analyses focus on connecting the professional characteristics of prin- 
cipals with school characteristics across the Dinaric region. We examined the rela- 
tionship of these characteristics with student achievement to explore differences and 
similarities across the region. 

Our review of the TIMSS 2019 regional data was designed to address four critical 
questions: 


(1) What are the licensing or certification requirements for principals across the 
Dinaric region? 

(2) What is the qualification level of principals? Is there an association between 
student mathematics and science achievement and principals' education levels 
across the Dinaric region? 

(3) How many of years of professional experience do principals across the region 
have generally? Is there an association between student mathematics and 
science achievement scores and this experience? 


! This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 
(United Nations, 1999) and the International Court of Justice (ICJ) Opinion on the Kosovo 
declaration of independence (ICJ, 2010). 
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(4) Do school characteristics (school location, school composition by socioeco- 
nomic background, and school emphasis on academic success) differ among 
Dinaric education systems? Are these characteristics related to variation in 
student achievement? 


2 Variables 


For our research analyses, we selected relevant data collected by the TIMSS 2019 
student school questionnaires, the latter completed by school principals (Table 1; for 
more details on the questionnaires, see TIMSS & PIRLS International Study Center, 
2018). We analyzed the associations between these variables and student achieve- 
ment scores in the TIMSS 2019 mathematics and science assessments using basic 
and advanced methods to estimate percentages, means, correlations, and develop 
regression models. We conducted statistical computations using established stan- 
dard procedures for handling data from large-scale assessments (see Sect. 5 for more 
details on the data sources, and the analysis methods and tools that we used). 

In addition to the data collected by the TIMSS 2019 study, we collected informa- 
tion about specific requirements for principals across the region by preparing a short 
additional questionnaire that we distributed to national research coordinators in the 
Dinaric region. We used this questionnaire to collect information on: 


relevant qualifying criteria for school principals; 

certification for school principals; 

models of professional development for school principals; and 

number of years that school principals or school directors have held their mandate. 


3 Results 


3.1 Characteristics of School Principals and Relation 
to Student Achievement 


In the Dinaric region, principals generally required at least five to eight years 
work experience in the education sector after acquiring their teaching degree before 
becoming a school principal, except in Kosovo where only three years of work expe- 
rience were required (Table 2). By law, in Montenegro, Bosnia and Herzegovina, 
North Macedonia, and Kosovo, school principals require a recognized university 
degree. In Croatia, school principals require a posteraduate university degree, with a 
total of at least eight years of work experience in schools or other institutions in the 
education system or in state administration bodies responsible for education (where 
at least five years should be acquired in education in school institutions). Albania, 
Montenegro, North Macedonia, and Serbia have also implemented a model of profes- 
sional development for school principals. Principals must complete a specialized 
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Table 1 List of variables and scales used in our analyses 
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Variable Description 


Values/response 
options 


Reference 


Principals’ years of | Years of experience as 


Number of years 


Fishbein et al. (2021) 


experience a principal Supplement 1, p. 100 
Principals' The highest completed | Did not complete Fishbein et al. (2021) 
education level of principals’ bachelor’s or Supplement 1, p. 100 


formal education 


equivalent level 
Bachelor’s or 
equivalent level 
Master’s or equivalent 
level 

Doctor or equivalent 
level 


School location Size of population in 


area of school location 


More than 500,000 
people 

100,001 to 500,000 
people 

50,001 to 100,000 
people 

30,001 to 50,000 
people 

15,001 to 30,000 
people 

3001 to 15,000 people 
3000 people or fewer 


Fishbein et al. (2021) 
Supplement 1, p. 95 


The scale is based on 
thirteen items that 
measure the principal’s 
perception of students’, 
parents’ and teachers’ 
focus on student 
achievement 


School emphasis on 
academic success 
scale? 


Higher values represent 
more emphasis on 
academic success 


Martin et al. (2020) 
p. 16.124 


Based on students’ and 
parents” reports 
regarding the 
availability of five 
resources: 

* Number of books in 
the home (students) 
Number of home 
study support 
(students) 

* Number of children's 
books in the home 
(parents) 

Highest level of 
education of either 
parent (parents) 
Highest level of 
occupation of either 
parent (parents) 


Home resources for 
learning scale? 


Higher values mean 
more home resources 
Index: Many resources, 
Some resources, Few 
resources 


Martin et al. (2020) 
p. 16.39 


(continued) 
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Table 1 (continued) 
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Variable 


School composition 
by socioeconomic 
background 


Description 


Principal’s report on 
the share of students in 
the school coming 
from: 
* Economically 
disadvantaged homes 
* Economically 
affluent homes 
Response options: 
0-10%, 11-25%, 
26-50%, more than 
50% 


Values/response 
options 


More affluent: Schools 
where more than 25% 
of the student body 
comes from 
economically affluent 
homes and not more 
than 25% from 
economically 
disadvantaged homes 
More disadvantaged: 
Schools where more 
than 25% of the student 
body comes from 
economically 
disadvantaged homes 
and not more than 25% 
from economically 
affluent homes 

Neither more affluent 
nor more 
disadvantaged: All 
other possible response 
combinations 


Reference 


Fishbein et al. (2021) 
Supplement 3, p. 19 


Note “These TIMSS scales are constructed so that the scale center point of 10 is located at the mean 
score of the combined distribution of all TIMSS 2019 grade four participants. The units of the scale 
are chosen so that the standard deviation of the distribution corresponds to two scale score points. 
For more information on scale construction, please see Yin and Fishbein (2020) 


Table 2 Overview of qualifications required for a school principal position in the Dinaric region 


Education Minimum years | Does principal Is there an Number of years 
system ofexperience in | need to complete a | established model | that the school 
education specialized school | of professional principal 

leadership training | development for | appointment lasts 
program school principals? 

Albania Yes Yes Unlimited 

Bosnia and No No 4 

Herzegovina 

Croatia 8 No No 5 

Kosovo EN Yes No 4 

Montenegro Yes Yes 4 

North Yes Yes 4 

Macedonia 

Serbia 8 Yes Yes 4 


Note *In 2019, this changed from three to four years 
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school leadership training program in Albania, Kosovo, Montenegro, North Mace- 
donia, and Serbia; in Albania, Montenegro, North Macedonia, and Serbia, successful 
completion of this program gives school principals a license, which indicates that 
they have met the required level of general and professional competencies. In all 
education systems in Dinaric region, the school principal is selected after an open 
call has been issued, and, in general, school principals in the Dinaric region are 
appointed for a term of four years, although, in Croatia, their appointment is for five 
years and, in Albania, for an unlimited period. In Croatia and Serbia, principals can 
be reappointed multiple times; in Serbia, their prior position in school is held for two 
terms. In Bosnia and Herzegovina, Kosovo, Montenegro, and North Macedonia, the 
school principal can be appointed to a school no more than twice consecutively. All 
this is regulated by national legislation. 

Across the Dinaric region, Croatia reported the highest percentage of school prin- 
cipals with a master’s or a doctorate degree level qualification (96%), followed by 
Albania (51%) (Table 3). In most other systems in the region, the majority of schools 
are managed by a principal holding a degree at bachelor level or an equivalent (Table 
3). 

For each level of principal qualification, we also analyzed the related percentages 
of students and their achievement differences in mathematics and science (see Table 
S.14 in the supplementary materials available for download at www.iea.nl/public 
ations/RfEVol13). We found that the association between student achievement and 
principal education was only significant in Montenegro, where grade four students 
in schools where principals had completed a postgraduate degree tended to have a 
significantly higher achievement than students in schools where principals had only 
completed a bachelor's degree or equivalent. They obtained, on average, 16 score 


Table 3 Percentage of principals by educational level 


Education system Percentage of school principals who: 
Did not complete | Completed bachelor's | Completed 
bachelor's degree | degree or equivalent | postgraduate 
but not a postgraduate | university degree 
degree 
Albania (1.6) 45 (4.8) 51 (5.0) 
Bosnia & Herzegovina (1.9) TI (3.3) 16 (3.0) 
Croatia (0.6) 3 (2.8) 96 (2.9) 
Kosovo? 14 (3.8) 53 (5.3) 33 (5.6) 
Montenegro 7 (2.3) 78 (3.5) 15 (2.8) 
North Macedonia 1 (0.8) 78 (3.9) 21 (4.0) 
Serbia?! 2 (1.6) 65 (5.5) 33 (5.5) 


Notes Standard errors appear in parentheses 

? National defined population covers 90-95% of the national target population 

lFor Serbian principals that graduated after 2005, a postgraduate master's degree is obligatory, 
according to Serbian law (Teodorovié et al., 2019) 
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points more in the TIMSS grade four mathematics test and 14 score points more in 
the TIMSS grade four science test. 

One of the primary development strategies for principals is experience acquired 
during the course of their work, with a general expectation that principals become 
more effective with increased experience working in a position of that level. On 
average, across the Dinaric region, principals had less experience than the inter- 
national TIMSS average (10 years); principals in North Macedonia had notably 
low levels of experience, while Serbia reported the highest average (Fig. 1; for a 
more detailed analysis, see Table S.15 in the supplementary materials available for 
download at www.iea.nl/publications/RfEVol13). 

When we analyzed the percentage of students by their principal's number of 
years of experience, we found that, in North Macedonia, a staggering 71 percent 
of students were learning in primary schools managed by principals that had less 
than five years of experience (Fig. 2). This percentage was also significant in Kosovo 
(34%) and Bosnia and Herzegovina (57%). While Albania had the lowest percentage 
of principals with less than five years of experience, 28 percent of grade four students 
still had a principal with little experience of the role. The TIMSS international average 
was 31 percent; students in the Dinaric region are thus more likely to have less 
experienced principals than students in other parts of the world. Looking at the same 
topic from another angle, only about nine percent of students in North Macedonia 
learn in schools managed by principals with at least 10 years of experience, which 
is lower than in other systems in the region (13% in Bosnia and Herzegovina, and 
>30% in Croatia, Kosovo, Serbia, and Montenegro). 

However, we found that the TIMSS 2019 grade four data for the Dinaric region 
did not provide any evidence of a relationship between student achievement and prin- 
cipals’ work experience. Calculating the correlation coefficient between the number 
of years of experience as a principal with the achievement of the students yielded 


Fig. 1 Average number of 9.0 
years of experience of school 


cad 8.0 77 
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Note In Kosovo and Serbia, 7.0 - 
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Albania 49 Principals' experience: 
Bosnia & Herzegovina 12 Wiss ysars 
Croatia 28 T e ^ 25 years to < 10 years 
Kosovo 30 2 10 years to « 20 years 
Montenegro 24 EJ m >20years 

North Macedonia 7 

Serbia 31 

0 20 40 60 80 100 


Percentage of students (%) 


Fig. 2 Percentage of grade four students by their principal's number of years of experience 
Note In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population 


no statistically significant results. Earlier research on this topic is ambiguous. Our 
findings are in agreement with some studies, where the data suggested that the 
experience of principals had no close relationship with academic achievement of 
students (Brockmeier et al., 2013; Gentilucci & Muto, 2007). In contrast, Dhuey and 
Smith (2014) reported that principal characteristics had significant effects on student 
achievement in mathematics and reading, and identified a weak relationship between 
principals’ levels of education and student test scores. 


3.2 Characteristics of School 


A variety of factors contribute to student achievement in mathematics and science, 
including student behaviors and student, teacher, and school characteristics. We 
focused on the school characteristics of school location, school composition by 
socioeconomic background, and school emphasis on academic success. 

When comparing education systems across the Dinaric region, formative charac- 
teristics are important for contextualizing the findings. The number of people who 
live in the city, town, or area where the school is located may have an impact on 
students achievements in the Dinaric region. The Dinaric region generally has low 
levels of urbanization, and at least two-thirds of the region’s students attend schools 
located in villages or small towns. However, there is still considerable variation in 
school locations across the region (Fig. 3). In Albania and Kosovo, a third of students 
attend a school located in an area with 3000 people or less, while, conversely, nearly a 
third of students in Serbia learn in schools located in a place with more than 100,000 
people (Fig. 3). 

We investigated whether student achievement was related to school location by 
comparing the TIMSS mathematics and science achievement of grade four students 
in schools in areas with >30,000 people to that of grade four students in schools in 
areas with <30,000 people (Table 4). Our results indicated that students from bigger 
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Fig. 3 Percentage of students by school location 
Note In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population 


Table 4 Mathematics and science achievement difference by school location 


Education system Difference in student achievement 
Mathematics achievement Science achievement 

Albania 32 (7.2) 34 (7.1) 
Bosnia & Herzegovina 14 (5.6) 14 (5.8) 
Croatia 13 (4.4) 13 (3.5) 
Kosovo? —6 (7.0) —4 (8.5) 
Montenegro 2 (3.1) -1 (3.5) 
North Macedonia 31 (11.8) 39 (12.9) 
Serbia? 38 (5.4) 37 (5.7) 


Notes Positive values indicate higher achievement in areas with 230,000 people compared to areas 
with «30,000 people. Statistically significant (p « 0.05) differences are shown in bold. Standard 
errors appear in parentheses 

* National defined population covers 90-95% of the national target population 


cities achieved higher scores than their peers in smaller cities or rural areas; the 
achievement differences for both mathematics and science were significant across 
most of the Dinaric region, except for Kosovo and Montenegro. This difference was 
most pronounced in Albania, North Macedonia, and Serbia, where it exceeded 30 
points for both mathematics and science (this corresponds to one-third of the standard 
deviation of the achievement scale score metric). 

There may be multiple reasons for similar clustering effects in the Dinaric region, 
for example, social segregation of residential areas (in combination with the tendency 
for children to attend nearby schools), fees for schools, lack of incentives for teachers 
to elect to work in more challenging areas, and/or better equipment in affluent schools 
because parents make additional financial contributions. 

The social background of families is often reflected in the student intake of schools. 
Parents with similar backgrounds tend to send their children to schools where they 
will meet similar children (Cahill, 2009); this can boost the school-level effect on 
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learning because children in more affluent schools may already start school with 
a higher “knowledge baseline” as a consequence of parental factors. We used the 
school principal responses about the socioeconomic background of the student body 
to group schools into three categories: “more affluent,” “neither more affluent nor 
more disadvantaged,” and “more disadvantaged” (Fig. 4). 

We noted that students from different backgrounds seemed to be strongly segre- 
gated in many schools in the Dinaric region (Fig. 4). The high percentage of students 
learning in disadvantaged schools in Albania is particularly noteworthy (almost half 
of Albania’s grade four students attend such schools). In Croatia, the comparable 
percentage was only 13 percent, while 57 percent of grade four students were reported 
as attending more affluent schools. In North Macedonia, only 10 percent of students 
attended “neither more affluent nor more disadvantaged” schools, but 66 percent 
attended more affluent schools. 

We compared average student achievement scores for grade four students from 
schools with a more affluent student body with those of students from schools with 
a more disadvantaged student body (Fig. 5). The results indicated that students in 
more affluent schools tended to achieve higher mathematics and science scores in 
TIMSS 2019. These differences were generally significant, except in Bosnia and 
Herzegovina; in Kosovo, only the difference in mathematics achievement was signif- 
icant. In Croatia, Montenegro, and Serbia, the difference in both subjects was around 
20 points, but, in Albania and North Macedonia, the difference exceeded 39 score 
points, amounting to almost half a standard deviation of the scale metric. 

School emphasis on academic success generally plays an important role in 
supporting or stimulating students in their learning, and implies effective teaching, 
a motivated work environment, and high levels of expectation for student success. 

In TIMSS 2019, principals were asked to report on their school’s emphasis on 
academic success. The TIMSS scale “school emphasis on academic success” (SEAS) 
(see Table 1) is related to a number of similar items aimed at measuring aspects of 
the school’s emphasis on academic success and the degree of support offered by 


Socioeconomic background 
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Fig.4 School composition by principals” assessments of the socioeconomic background of the 
school’s student body 

Notes In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population. In Kosovo, data were available for >50% but <70% of the students 
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Fig.5 Difference in grade four student achievement scores by school socioeconomic background 
Notes ` Statistically significant (p < 0.05) differences. Positive values indicate higher achievement in 
more affluent schools compared to more disadvantaged schools. In Kosovo and Serbia, the national 
defined population covers 90-95% of the national target population 


the school organization and the school environment. Internationally, students who 
were classified as attending a school with “high or very high emphasis on academic 
success” are those with a scale score greater than or equal to 9.2. Students who were 
classified as attending a school with “medium emphasis on academic success” had 
a score lower than 9.2 (Table 5). 

Overall, more than half of students in the region attended schools where principals 
reported there was a “very high” or “high” emphasis on academic success as part 
of the school’s culture (Fig. 6). The average scale score across the Dinaric region 
ranged from 9.3 in North Macedonia to 10.3 in Montenegro (Table 5). 

We analyzed the correlation between SEAS and grade four student achievement 
across the region (Table 5). This correlation was very weak, but positive in all partici- 
pating education systems in the region for both subjects. The correlation coefficients 
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Fig. 6 Percentages of students attending schools with different emphasis levels on academic 
success, based on principals” assessments of their school culture 
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Table 5 School emphasis on academic success as reported by principals, and its correlation with 
grade four student achievement in TIMSS 2019 


Education system Average scale Correlation with Correlation with 
score of the mathematics science achievement 
TIMSS 2019 achievement 


school emphasis 
on academic 
success scale 


Albania 10.2 (0.1) 0.21 (0.04) 0.22 (0.05) 
Bosnia & Herzegovina 9.8 (0.1) 0.02 (0.04) 0.03 (0.03) 
Croatia 10.0 (0.1) 0.06 (0.04) 0.07 (0.04) 
Kosovo? 10.1 (0.1) 0.13 (0.04) 0.15 (0.04) 
Montenegro 10.3 (0.0) 0.02 (0.02) 0.03 (0.02) 
North Macedonia 9.3 (0.2) 0.21 (0.06) 0.23 (0.06) 
Serbia? 9.6 (0.1) 0.14 (0.05) 0.14 (0.05) 


Notes Statistically significant (p « 0.05) correlation coefficients are shown in bold. Standard errors 
are provided in parentheses 
National defined population covers 90-95% of the national target population 


were statistically significant in Albania, Kosovo, North Macedonia, and Serbia. This 
is in line with Bandura (1993) and Hoy et al. (2006) who confirmed a strong rela- 
tionship between academic optimism and student achievement; they further stated 
that efficacy, trust, and positive academic emphases together produce a powerful 
force that engenders motivation, creates collective optimism, and channels student 
behavior toward the accomplishment of high academic goals. Schools with academic 
optimism create collective beliefs that changes are possible and all students can learn, 
inspiring a confidence that high academic performance can be achieved. 


4 Conclusions 


We analyzed the characteristics of principals and schools in the Dinaric region and 
the relationship of these characteristics with academic achievement of grade four 
students in TIMSS 2019. The focus of any principalis to manage the various processes 
in the school and support the professional activities of their teachers to create a 
successful learning environment for students. While we cannot address all aspects 
of a principal's remit in our analyses, the TIMSS 2019 data provide useful informa- 
tion on their educational levels and number of years of experience as a principal, 
two factors that have been previously linked to student academic achievement (see 
Dhuey & Smith, 2014). While we were unable to identify a statistically significant 
relationship between these characteristics and grade four student mathematics and 
science achievement in our analyses of the TIMSS 2019 data, the role of the prin- 
cipal is undoubtedly a critical component in student achievement. They can establish 
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a positive academic environment that embraces cognitive, emotional, and behavior 
elements; the positive interaction of all these elements creates a school culture of 
academic success. 

It is essential that the management of the school creates an environment for 
learning and expects high achievement from the students because this facilitates 
and improves the achievement of the students. School principals should establish an 
environment and culture where all involved parties contribute toward supporting and 
improving student achievement. 

There are also non-malleable factors that shape the learning environments of 
students. One of them is school location. We found that the level of urbaniza- 
tion of the area surrounding the school can also be related to student achievement. 
We observed that students in areas with more people demonstrated higher achieve- 
ment scores in mathematics and science than students from schools located in less 
populated areas in Albania, Bosnia and Herzegovina, Croatia, North Macedonia, 
and Serbia (for further investigation into differences between urban and rural areas, 
see chapter “Scaffolding the Learning in Rural and Urban Schools: Similarities and 
Differences”). Conditions to promote students’ learning tend to be better in urban 
schools, and this tendency is also reinforced by conditions for learning at home, 
which significantly correlates with higher student achievement. 

The composition of the socioeconomic background of the student body of the 
school could also be related to grade four student mathematics and science achieve- 
ment. In general, we found that children from more disadvantaged environments 
scored lower TIMSS achievement than students from more affluent schools. These 
differences in student achievement were statistically significant in Albania, Croatia, 
Kosovo (only for science), Montenegro, North Macedonia, and Serbia. 

While, across the Dinaric region, TIMSS 2019 data showed no significant relation- 
ships between principal characteristics and student achievement, we caution against 
underestimating the importance of principals. Although we were unable to empiri- 
cally prove such relationships, this does not necessarily mean they do not exist, as 
there may be other characteristics associated with achievement that are not reported 
by TIMSS, and they may be interrelated and interdependent. 

An important indicator in this analysis was school emphasis on academic success 
(as reported by school principals). We found that the correlation between the school 
emphasis on academic success (as reported by their principals) and grade four student 
mathematics and science achievement was positive for all participating systems in 
the region, and statistically significant in Albania, Kosovo, North Macedonia, and 
Serbia. It is thus important that schools in the Dinaric region continue to promote an 
emphasis on academic success. School communities (principals, teachers, parents, 
and students) need to focus on working together to create a positive school climate 
that helps to establish students’ confidence in their abilities and motivate them to 
better achievement. 


Characteristics of Principals and Schools in the Dinaric Region 189 


References 


Alhosani, A., Singh, S., & Nahyan, M. (2017). Role of school leadership and climate in student 
achievement. International Journal of Education Management, 31(2), https://doi.org/10.1108/ 
IJEM-05-2016-0113. https://www.researchgate.net/publication/3 18254863 

Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational 
Psychologist, 28(2), 117—148. https://doi.org/10.1207/s15326985ep2802_3 

Brockmeier, L. L., Starr, G., Green, R., Pate, J. L., & Leech, D. W. (2013). Principal and school- 
level effects on elementary school student achievement.. International Journal of Educational 
Leadership Preparation, 8, 49-61. https://files.eric.ed.gov/fulltext/EJ1013001.pdf 

Cahill, R. (2009). Factors that influence the decisions parents make when choosing a secondary 
school for their children (Doctoral thesis, Edith Cowan University, Joondalup, Australia). https:// 
ro.ecu.edu.au/theses/549 

Dhuey, E., & Smith, J. (2014). How important are school principals in the production of student 
achievement? Canadian Journal of Economics, 47(2), 634—663. 

Fishbein, B., Foy, P., & Yin, L. (2021). TIMSS 2019 user guide for the international database. 
TIMSS & PIRLS International Study Center, Boston College. https://timssandpirls.bc.edu/tim 
ss2019/international-database/ 

Gentilucci, J. L., & Muto, C. C. (2007). Principals’ influence on academic achievement: the student 
perspective. NASSP Bulletin. 91(3), 219-236. https://doi.org/10.1177/0192636507303738 

Hoy, W. K. (2012). School characteristic that make a difference for the achievement of all students: 
A 40-years odyssey. Journal of Educational Administration, 50, 76-97. 

Hoy, W. K., Tarter, C. J., & Woolfolk Hoy, A. (2006). Academic optimism of schools: A force for 
student achievement. American Educational Research Journal, 43(3), 425-446. 

ICJ. (2010). Accordance with International Law of the Unilateral Declaration of Independence in 
Respect of Kosovo, Advisory Opinion, I.C.J. Reports 2010. International Court of Justice. https:// 
www.icj-cij.org/public/files/case-related/141/141-20100722-ADV-01-00-EN. pdf 

Malere, A., & Ozola, A. (2019), Role of school principals in high achievement of students. In 
Rural Environment. Education. Personality (REEP). Proceedings of the International Scientific 
Conference. Volume 12, 10-11 May 2019, Jelgava, Latvia (pp. 83—93). Latvia University of Life 
Sciences and Technologies, Faculty of Engineering, Institute of Education and Home Economics. 
http://bit.]ly/REEP. 2019 proceedings 

Martin, M. O., von Davier, M., & Mullis, I. V. S. (Eds.). (2020). Methods and procedures: TIMSS 
2019 technical report. TIMSS & PIRLS International Study Center, Boston College. https://tim 
ssandpirls.bc.edu/timss2019/methods 

OECD. (2013). What makes urban schools different? PISA in Focus, No. 28. OECD Publishing. 
https://doi.org/10.1787/5k4618w342jc-en 

Osborne-Lampkin, L., Folsom, J. S., & Herrington, C.D. (2015). A systematic review of 
the relationships between principal characteristics and student achievement (REL 2016— 
091). US Department of Education, Institute of Education Sciences, National Center for 
Education Evaluation and Regional Assistance, Regional Educational Laboratory South- 
east. https://www.researchgate.net/publication/286446967 A systematic review of the relati 
onships between principal characteristics and student achievement 

TIMSS & PIRLS International Study Center. (2018). TIMSS 2019 context questionnaires. TIMSS & 
PIRLS International Study Center, Boston College. https://timssandpirls.bc.edu/timss2019/que 
stionnaires/index.html 

Teodorović, J., Ševkušić, S., Malinić, D., & Đelić, J. (2019). Leadership in education: The case of 
Serbia. In S. Ševkušić, D. Malinić, & J. Teodorović (Eds.), Leadership in education: Initiatives 
and trends in selected European countries (pp. 163—180). Institute for Educational Research, 
Jagodina/Faculty of Education, University of Kragujevac/Hungarian-Netherlands School of 
Educational Management, University of Szeged. 


190 B. Lameva et al. 


United Nations. (1999). Resolution 1244 (1999). Adopted by the Security Council at its 4011th 
Meeting, on 10 June 1999. United Nations Security Council. https://digitallibrary.un.ore/record/ 
2/74488?Inzen 

Yin, L., & Fishbein, B. (2020). Creating and interpreting the TIMSS 2019 context questionnaire 
scales. In M. O. Martin, M. von Davier, & I. V. S. Mullis (Eds.), Methods and procedures: TIMSS 
2019 technical report (pp. 16.1—16.331). TIMSS & PIRLS International Study Center, Boston 
College. https://timssandpirls.bc.edu/timss2019/methods/chapter-16.html 


Beti Lameva is the head of the Sector for Exams, IT and Research at the National Examina- 
tion Center in the Republic of North Macedonia. She has more than 20 years of experience in 
education research, establishing large-scale assessment and high-risk exams, data entry, cleaning, 
and processing. Beti has also been involved in international studies since TIMSS 1999 as a data 
manager. She is currently the national research coordinator for the international studies, TIMSS 
and PISA. 


Zaneta DZumhur works for the Agency for Preschool, Primary and Secondary Education, since 
2009. She has intense experience in external evaluation at national and international levels. Her 
professional interests are divided between work on research of student achievement and devel- 
opment of learning outcomes. She is author and co-author of a variety of publications and arti- 
cles. She is also interested in closer cooperation between schools and different actors in school 
enviroments. 


Mojca RoZman is a research analyst at IEA's Research and Analysis Unit. Her background is in 
psychology and statistics. She has experience in questionnaire development and scaling of ques- 
tionnaire data. Her interests are methodology and statistical analysis in international large-scale 
assessments. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution- 
NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which 
permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium 
or format, as long as you give appropriate credit to the original author(s) and the source, provide a 
link to the Creative Commons license and indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Characteristics of High- (f 
and Low-Performing Students ess 


Rezana Vrapi, Agim Alia, and Falk Brese 


Abstract Many governments are interested in improving the overall attainment of 
their school students and in delivering quality education for all that improves the life 
opportunities of their populations. In addition to comparing average student achieve- 
ment with similar economies, looking in depth at the factors that affect variation in 
student performance and underlie student achievement gaps can provide important 
information to support educational improvement. Students that find it difficult to 
perform even basic mathematical computations or understand elementary scientific 
concepts may be left behind if they do not receive specific help in the early years 
of education to lay the foundations for later school years. At the same time, it is 
also important to foster the talents of students that are gifted in mathematics and 
science, as this group are likely to become an important part of the future work 
force. IEA's Trends in International Mathematics and Science Study (TIMSS) can 
be used to analyze aspects of student achievement and the background factors that 
influence how students learn about mathematics and science. Such data can be used 
to evaluate the proportions, competencies, and characteristics of these groups of 
high- and low-performing students across the Dinaric region. The competencies of 
the two groups can be established by analyzing student proficiency levels relative 
to the TIMSS international benchmarks in mathematics and science. Analyzing the 
characteristics of these high- and low-performing students revealed that there were 
considerable differences in the proportions of grade four students lying at either end 
of the TIMSS achievement distribution across the Dinaric region. For mathematics, 
boys tended form a higher proportion of the group of high-achieving students in 
three of the Dinaric systems, but conversely, in science, boys were more often found 
in the low-achieving group in three systems. The availability of home resources for 
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learning varied significantly across the participating Dinaric education systems and 
was found to be positively related to student attainment. Student attitudes towards 
learning the subjects and student reports of their physical wellbeing on arrival at 
school were also found to be related to student achievement across the region. 


Keywords Dinaric region : High achievers * Low performers : Mathematics 
instruction * Primary education : Science instruction * Trends in International 
Mathematics and Science Study (TIMSS) 


1 Introduction 


Raising the number of high performing students and reducing the proportion of low 
performers is considered an important educational goal for every country. Fostering 
higher order thinking among students of all ages is considered another critical educa- 
tional objective. However, teachers often believe that this latter ambition is not 
intended for or applicable to all their students; a common belief among teachers 
is that tasks requiring higher order thinking are appropriate only for high-achieving 
students, whereas low-achieving students, who can barely master the basics, are 
unable to deal with such tasks (Zohar et al., 2001). We examined the proportions and 
characteristics of the two groups of students identified as the high and low performers 
across the Dinaric region. Our aim was to identify the obstacles related to their perfor- 
mance, in order to understand what teaching strategies or changes in the education 
system might best support their learning and achievement. The achievement of those 
students should be viewed as the result of their efforts, despite the barriers that might 
impede their performance. Our second objective was to provide evidenced analyses 
of the regional goals related to these groups of high- and low-achieving students, 
with the aim of helping the Dinaric education systems to identify practical measures 
that support both teachers and other stakeholders in achieving desired results. While 
the average achievement of students in an education system is interesting, investi- 
gating the extremes of the achievement distribution in more depth carries the potential 
to identify tailored and distinct solutions to support both academic excellence and 
those students who are struggle with the basic concepts of mathematics or science 
(see Meinck & Brese, 2019). 

Seven participants from the Dinaric region took part in TIMSS 2019, namely 
Albania, Bosnia and Herzegovina, Croatia, Kosovo,! Montenegro, North Macedonia, 
and Serbia. Data from the 2019 cycle of IEA's Trends in International Mathematics 
and Science Study (TIMSS) thus provide a unique opportunity to study the mathe- 
matics and science performance of these two groups of grade four students across 
the Dinaric region. Our initial research questions were: 


! This designation is without prejudice to positions on status, and is in line with UNSCR 1244/1999 
(United Nations, 1999) and the International Court of Justice (ICJ) Opinion on the Kosovo 
declaration of independence (ICJ, 2010). 
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(1) Across the Dinaric region, what percentage of students can be categorized 
as high achievers? What percentage of students can be considered the low 
achievers? 

(2) What are the characteristics of high- and low-performing students? Do 
these characteristics differ across the region, and, if so, what causes these 
characteristics to differ? 


Students performing at the top level of academic achievement demonstrate a 
deeper understanding of a subject than their peers and can apply their skills and 
knowledge to more complex situations. In examining the differences between these 
two groups of different competencies, it is important to question why there are these 
differences both within and across education systems; differences between education 
systems may reflect their diversity, and differing strengths and weaknesses. 

A vast amount of literature provides evidence that differences in student achieve- 
ment are related to many factors (Atar and Atar, 2012; Aypay et al., 2007; Papanasta- 
siou, 2008; Papanastasiou & Papanastasiou, 2004; Papanastasiou et al., 2004; Yayan 
& Berberošlu, 2004). According to Mullis et al. (2020), variables related to home 
background, resources at school and school climate, teaching methods, and students’ 
attitudes towards learning and towards the subjects are all significantly related to 
student achievement in many countries around the world. 

According to the TIMSS reports, some countries have a considerable proportion of 
students that perform at an academically advanced level, while others do not, which 
naturally leads educational policymakers and researchers to question why such differ- 
ences arise. Understanding the policies and practices that lead to high-quality learning 
outcomes is clearly valuable, and many studies have investigated the nature of the 
relationship between students' attitudes and their achievement (Atar and Atar, 2012; 
Aypay et al., 2007; Ceylan & Berberoglu, 2007). Student attitudes toward science 
were identified as being significantly positively correlated with science achieve- 
ment (Papanastasiou et al., 2004). Gibson and Chase (2002) found that activities 
that invited students to actively engage in science using a hands-on inquiry-based 
approach helped middle school students to develop an interest in science that they 
tended to maintain during their years in high school education. Thus, strong science 
reasoning scores and positive attitudes toward science in high-performing schools 
may be partially attributed to the type of implemented instructional practices used 
in the science classrooms in these schools. 

A student's socioeconomic status (SES), which is also generally related to 
the educational background of their family members, has been identified as a 
factor that may also be related to their school performance (Papanastasiou, 2008). 
Although researchers may define SES in slightly different ways, robust relationships 
between student SES and test scores have been well replicated by social scientists 
(Konstantopoulos, 2005; White et al., 1993). 

However, it is not always easy to determine definitely which factors make the crit- 
ical difference. We thus opted to focus on student-centered factors rather than system- 
level factors or policies; these include gender, home resources for learning, student 
attitudes towards mathematics and science, and their assessment of their physical 
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ability to attentively follow lessons at school. Gender, home resources for learning, 
and (positive) attitudes towards the subject are characteristics that are frequently and 
regularly checked for their associations with student achievement (see, for example, 
Mullis et al., 2020). Home resources for learning are sometimes used as a proxy for 
the wealth and/or social status of the student’s family. Gender equity is perceived as 
a universal goal, and therefore one of the major aims of the sustainable development 
goals set by the United Nations (2018). Students’ positive attitudes toward learning 
mathematics and science have been shown as strongly related to academic achieve- 
ment in those subjects (Mullis et al., 2020). Finally, the physical well-being of the 
student has been hypothesized as having an effect on their achievement; research has 
shown that the students that reported getting more hours sleep that their peers also 
tended exhibit significantly less daytime-sleepiness-related behaviors (Owens et al., 
2010). Recently, Lin et al. (2020) found a direct association between reported sleep 
duration and the mathematics achievement scores of adolescent female students. 


2 Data and Methods 


The TIMSS assessment sets four benchmark levels for both mathematics and science 
achievement (Mullis et al., 2020) depending on student performance on the TIMSS 
mathematics and science test. These benchmarks are defined in terms of cut points 
on the continuous achievement scale as follows: “advanced” (students scoring at or 
above 625 points), “high” (students at or above 550 and below 625 score points), 
“intermediate” (students at or above 475 and below 550 score points), and “low” 
(students at or above 400 and below 475 score points). For example, a student scored 
460 points on the mathematics test is categorized as having reached the TIMSS low 
international benchmark of mathematics achievement (at or above 400 and below 475 
score points). A student whose performance was more than one standard deviation 
below the scale center point (i.e., below 400 points) is described as not reaching the 
TIMSS low international benchmark. 

These benchmarks provide a simplified picture of the variation in student achieve- 
ment across different educational systems. Our focus was on the grade four students 
at both ends of the achievement distribution in mathematics and science. Thus, 
according to the performance benchmarks determined by TIMSS, we distinguished 
two groups for our research analyses: the students that fell below the TIMSS low 
international benchmarks (scoring less than 400 score points, hereinafter referred 
to as the low achievers), and the students that scored at or above the TIMSS high 
international benchmark (550 or more score points, hereinafter referred to as the high 
achievers). For both groups, we computed and compared their proportions in each 
education system across the Dinaric region. 

We aimed to investigate whether a particular set of contextual factors was partic- 
ularly related to the achievement of these two groups. TIMSS administers a number 
of background context questionnaires. Responses to these questionnaires may be 
used to identify specific factors that seem to be related to high and low achievement 
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across the Dinaric region, providing important evidence on the contexts of learning 
that can inform our analyses. We selected a number of variables and indices derived 
from data collected by the TIMSS 2019 student, teacher, and school questionnaires 
(Table 1), which we used to assess student gender (male/female), student access to 
home resources for learning, and their attitudes toward mathematics and science. 
In TIMSS 2019, students were also asked to assess how often they felt tired when 
they arrived at school. We used student responses to this question to assess whether 
their physical well-being on arrival at school was related to achievement in these two 
groups of students. 

We used simple statistical indicators, such as means and percentages, to describe 
the characteristics of the groups of high- and low-achieving students (please refer to 
Sect. 5 for further information on the data and methods used in our analyses). 


2.1 Benchmark Performance: Grade Four Mathematics 


Each of the four benchmarks in mathematics are defined by the typical skills displayed 
by students who reach a particular benchmark (see Mullis et al., 2020). Students at 
higher benchmarks show a better understanding of the respective subject and the 
ability to solve more complex problems than students at lower benchmarks. More 
specifically, students at the TIMSS low benchmark in mathematics could provide 
evidence of basic mathematical knowledge, while those at the high benchmark could 
solve increasingly complex problems using more advanced skills, particularly the 
ability to complete multi-step problems (Table 2). In our analyses, we focused on the 
students performing below the low benchmark and at or above the high international 
benchmark in mathematics. 


2.2 Benchmark Performance: Grade Four Science 


As with mathematics, the characteristics for the TIMSS international benchmarks 
for science at grade four define increasing levels of scientific knowledge and under- 
standing from the low to the advanced benchmarks (Mullis et al., 2020). Students 
at the low benchmark could demonstrate basic knowledge of the life sciences and 
physics, and were able to interpret simple tables and diagrams. Again, students 
that did not reach this benchmark failed to answer even simple questions related to 
the subject and had not understood very elementary natural concepts. By contrast, 
students at the high benchmark could communicate and apply knowledge of life, 
physical, and earth science concepts in everyday and abstract contexts (Table 3). 


R. Vrapi et al. 


196 


(pənunuoo) 


(6€'91 `d “0Z0Z) urequsr pue UIA 
sooualoJow 


soomoso1 MOT (€) 

səoInosəi 20106 (2) 

saoinosar AUEIN (T) 

I 8911089329 

30I} YM Xxopul UL olur so102s 

Mə o[vos Sutsn pƏptAIp SEM APLIS SYL 
ZuruIea 10} soo1noso1 

Əutou SIOW ULJU sən[0A I9YFIH 


suondo osuodsow/son[eA 


(syuored) juared 

Jeuro jo uonednooo Jo [oA9| 1SƏUSIH 
(syuored) mad 

Jogo Jo uoneonpo Jo [oA9[ 1SƏUSIH 
(syuored) ouioq 

Əu ur sxooq s,uəirplruo Jo JoquinN 
(sjuopnis) 

31oddns Apps əwoy jo Joquinyy 
(sjuopnjs) 

Əutou oq ur sxooq Jo 19quinN e 
:S9OINOSOI 

ƏAU Jo Aimqe[mae oy) Surpresar 
syiodal ,sjuo1ed pue ,sjuopnjs uo poseg 


uondriosoq 


yo [eos 
Sura 10] soo1nosa1 IWOH 


epeog/so[quueA 


sosÁTeue INO ur posn SAVIS pue so[qerrA 201 JOIST] TAQEL 


197 


(pənunuoo) 


(td 
‘q 1uəurə[ddnS *1707) ‘Te 19 urequstd 


kog 
HI 


sjuopnis ou Aq poprAoud 
JopuoS ,sjuopnjs uo UOT]EULIOJU] 


Jopuos juopng 


(96°91 d 
pue 6879] `d ‘OZOZ) urequsrd pue ux 
SIDUMA 


IDUIS JO soneuroqjeui , Sururpə[ 

JI 1euAƏuos,, sjuəpnis 110 [TV 
eSv1oA? UO ‘MOJ IIO ou) YM 

„OMU e Suroo18e,, pue sjuouirojejs ouru 
ou Jo VAY YIM 20111 L Zutaten, 

o) Surpuodsou100 3109S 1nd au ^o[oq 

IO 16 2109S ? pey 22uaros JO songurəu)pur 
,SuruTeo[ IJI JOU op,, OYM sjuopnig 
ovo? UO ‘MOJ IIO ou) 

YM SMY e guroo18e,, pue sjuouiojejs 
ouru 201 JO IAY YIM JOJ P Suroo1Sv,, 

0} Surpuodsar1oo 3109s 103 ay) oAoqe 

IO 16 9109S ? DEU oouoros/sorulotgeur 
,SuruTeo[ IJH YONU K1oA,, OYM sjuopnis 
exi Jou oq (£) 

exip YMOWIOS (c) 

exi yonu mn (1) 

S310393 oot] YIM xepug 


suondo osuodsos/son[eA 


syoalqns 2104) 

Kul JO euo ST oouoros/soneureuepA 
‘syuouttiodxe oouoros 

OP 0} IJI [/Suosso[ soruioqgeui oj 
pemo} YOO] [ 0A pom oy ur sSur 
MOY SUI səy e»uerog/sure[qo.d 
soneuiroujeui DATOS O] AN 

I 5[00u5s ur oouoros Surueo[ 0} peAJ0] 
oor [/s1equinu soA[oAur yey JIomjooyos 
Aue at I oouoros/soneuroteur 

exi] I feouoros/soneuroqeur 

ur sgum) Surso1ojur Augu uwa] J 
{BULIOG sr oouoros/soneurouye]q 
*oouoros/soneuroqyeui 

Apnys 0} ƏAtu 30u pip Į USIA 

I :eouoros/soneuroqyeur Sutures, Kofuo J 
:eouoros/sonvuroueur 

JUTU jnoqe sJuəutəlp1s 

Əuru o) juaurooJ8e ,sjuopms Jo o[eog 


uonduosoq 


yo [825 9ouoros/soreuroqeur 
Surueo[ IJH sjuopnjg 


epe»g/so[quueA 


Characteristics of High- and Low-Performing Students 


(penunuoo) I əlqeL 


R. Vrapi et al. 


198 


(OZOZ) ‘Te 1e UEIN oos oseo[d “səroos MI AVIS pue “Solopouəur FULIS *uononnsuo2 VIS uo uorjeuuojur [VIOUS IIOU JO | 
'sjutod 3109S Ə[03s OM} 0} spuodsauioo uonnqrnsip ou JO UORIAAP pipputp3s ou WY os uəsouo are FLIS au) JO sjrun AL 'sjuedronred 0j app 
6107 SSINLL II? Jo uonnqrnsip peurquioo oY) Jo oJoos uvoui ou Je p9162o| SI QT Jo 1urod 193uoo AVIS au Jey) os pə)onnsuo2o IE SAVIS SSIALL 9S9UL, $270N 


(91 d 
‘q ueurojddns *[ 707) ‘Te 19 urequstd 


soouoloJow 


JOAON (p) 
səouməuioç (ç) 


Kep K1oAo asoy (2) 
Kep Aroq (1) 


:Sorogojeo asuodsal INOA 


suondo osuodsos/son[eA 


oouos 1e Zume uouA parn Sulpooy 
jo Aouanbayy uo suodat ,syuopms 


uonduosoq 


uonipuoo juopnig 


epeog/so[quueA 


(penunuoo) TAQEL 


Characteristics of High- and Low-Performing Students 199 


Table2 Descriptions of the TIMSS 2019 high and low international benchmarks of mathematics 
achievement 


Benchmark Score Students typically 

High At least 550 TIMSS points | Can apply conceptual 

Students can apply conceptual understanding of whole numbers 
understanding to solve problems to solve two-step word problems 


Show understanding of the 
number line, multiples, factors, 
and rounding numbers, and 
operations with fractions and 
decimals 

Can solve simple measurement 
problems 

Demonstrate understanding of 
geometric properties of shapes 
and angles 

Can interpret and use data in 
tables and a variety of graphs to 
solve problems 


Low Below 400 TIMSS points | Can add, subtract, multiply, and 
Students have some basic divide one- and two-digit whole 
mathematical knowledge numbers 


Can solve simple word problems 
Have some knowledge of simple 
fractions and common geometric 
shapes 

Can read and complete simple 
bar graphs and tables 


Source Mullis et al. (2020, exhibit 1.7) 


3 Percentages of High- and Low-Achieving Students 


Percentages of grade four students at or above the TIMSS high international bench- 
mark and below the TIMSS low international benchmark in mathematics varied 
substantially across the Dinaric region (Fig. 1). Croatia has by far the lowest 
percentage of low-achieving students (5%); there were also relatively low propor- 
tions of low-achieving students in Serbia (11%) and Albania (14%). However, across 
the remainder of the region, about a quarter of students fell into this group, which 
clearly demands remedial action as there is potential for this achievement gap to 
further widen during subsequent education, permanently affecting students” future 
life opportunities. 

At the other end of the achievement distribution, the variation between systems 
was also substantial (Fig. 1). Serbia, Croatia, and Albania had the highest proportion 
of high achievers (i.e., students at or above the TIMSS high international benchmark 
in mathematics at grade four; 32%, 28%, and 26%, respectively). In all other systems 
across the region, only about a tenth of students mastered this level of achievement. A 
comparison between the percentages of low achievers and of high achievers showed 
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Table 3 Descriptions of the TIMSS 2019 high and low international benchmarks of science 


achievement 
Benchmark Score Students typically 
High At least 550 TIMSS points | Communicate knowledge of 
Students communicate and characteristics of plants, 
apply knowledge of life, animals, and their life cycles, 


physical, and Earth science 


and apply knowledge of 
ecosystems and of humans’ and 
organisms’ interactions with 
their environment 

Demonstrate knowledge of 
states and properties of matter 
and of energy transfer in 
practical contexts, and show 
some understanding of forces 
and motion 

Know various facts about the 
Earth’s physical characteristics 
and show basic understanding of 
the Earth-Moon-Sun system 


Low 

Students show limited 
understanding of scientific 
concepts and limited knowledge 
of foundational science facts 


Below 400 TIMSS points 


Can recognize that some animals 
have backbones, that some 
materials conduct heat better 
than others, and that water and 
soil are natural resources 


Source Mullis et al. (2020, exhibit 2.7) 
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W Ator above high benchmark 


At or above the low benchmark 
and below the high benchmark 


A 


Below low benchmark 


Fig. 1 Percentages of high- and low-achieving students in TIMSS 2019 mathematics. Note In 
Kosovo and Serbia, the national defined population covers 90-95% of the national target population 
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Fig. 2 Percentages of high- and low-achieving students in science TIMSS 2019. Note In Kosovo 
and Serbia, the national defined population covers 90-95% of the national target population 


that there were more high achievers than low achievers in only three participants: 
Croatia (2396 more), Serbia (2196 more), and Albania (1296 more). In North Mace- 
donia, the percentages of both groups were similar. The percentage of low achievers 
was higher than the percentage of high achievers in Bosnia and Herzegovina (1596 
higher), Kosovo (19% higher) and Montenegro (13% higher). These data indicate 
that the education systems of the Dinaric region need to consider action, as it is 
possible that nations with low proportions of high achievers in their schools may 
subsequently lack sufficiently qualified staff in the workplace. 

Percentages of grade four students at or above the TIMSS high international 
benchmark and below the TIMSS low international benchmark in science also varied 
substantially across the Dinaric region (Fig. 2). It was an especially noteworthy 
achievement that few students in Croatia failed to achieve the low TIMSS bench- 
mark. Serbia also only had relatively few low achievers in science (896), followed by 
Albania (1496). Worryingly, however, nearly a fifth of students in Bosnia and Herze- 
govina (2296) seemed to find it difficult to answer questions on natural phenomena 
that should be familiar to grade four students. This problem was apparently even 
more severe in Montenegro (25% of students), North Macedonia (38% of students), 
and, finally, Kosovo (41%), where four out of ten students were categorized as low 
achievers (Fig. 2). 

Serbia and Croatia showed the highest percentages of students at or above the 
TIMSS high international benchmark in science (36% and 34% of their students were 
high achievers, respectively). There was quite a large apparent difference between 
these two systems and other systems across the Dinaric region. In Albania, 24% 
of students were high achievers, but the proportions of high achievers were much 
lower in Montenegro (1296), Bosnia and Herzegovina (1296), and North Macedonia 
(11%), while, in Kosovo, only four percent of students scored at or above high TIMSS 
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benchmark. Comparisons of the percentages of students below the low TIMSS bench- 
mark with those at or above the high TIMSS benchmark showed that, in Croatia, 
Serbia, and Albania, there were more high achievers than low achievers. In Kosovo, 
North Macedonia, Montenegro, and Bosnia and Herzegovina, there were more low 
achievers than high achievers (Fig. 2). 

Considering the proportion of students lying at the extreme ends of the achieve- 
ment distribution, it becomes evident that, across the region, fewer students perform 
at an intermediate level (i.e., at or above the low TIMSS benchmark but below the high 
TIMSS benchmark) in science than in mathematics. Hence, educational inequities 
seem to be more pronounced in science than in mathematics across the region. In 
comparison to other systems in the Dinaric region, Serbia and North Macedonia 
had higher proportions of students in these extreme performance categories in both 
subjects; this finding also suggests underlying issues of equity exist in these particular 
education systems. 


4 High and Low Achievers by Gender 


In general, across the seven Dinaric participants, gender did not seem to be signifi- 
cantly related to high or low student achievement in mathematics or science. There 
were either no gender gaps in the proportions of high and low achievers at grade 
four, or, in the few cases where gender gaps were noted, these were rather small and 
not of consequence. 

The high achievers in mathematics showed no gender gaps in four of the seven 
Dinaric systems. In three participants, proportionally more boys reached the TIMSS 
high international benchmark in mathematics than girls (Fig. 3). The largest differ- 
ence between the proportions of girls and boys at or above the high TIMSS benchmark 
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Fig. 3 Distribution of female and male students across the TIMSS international benchmarks in 
mathematics. Note In Kosovo and Serbia, the national defined population covers 90-95% of the 
national target population 
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Fig. 4 Distribution of female and male students across the TIMSS international benchmark in 
science. Note In Kosovo and Serbia, the national defined population covers 90-95% of the national 
target population 


was in Croatia (896). For the low achievers, only Serbia showed a significant differ- 
ence between the proportions of girls and boys, with a higher proportion of male 
students not reaching the low TIMSS benchmark. The other six participants showed 
no gender gaps for the students below the low TIMSS benchmark for mathematics. 

In general, TIMSS has consistently shown that boys have had an almost universal 
advantage in mathematics since the first cycle of TIMSS in 1995; the few exceptions 
have tended to be Middle Eastern and North African countries. In many countries, 
the gender gaps only increase between grades four to eight (Meinck & Brese, 2019), 
suggesting that gender gaps may increase over time if not tackled. 

The gender distribution of the low achievers in science suggested that, in most 
of the participating systems, higher proportions of boys than girls failed to achieve 
the TIMSS low international benchmark in science (Fig. 4). These differences were 
significant in Kosovo (6%), Montenegro (3%), and North Macedonia (6%). Albania 
was the only participant that showed a small but significant gender difference among 
the high achievers in science (396 in favor of girls). In all other participants, there 
were no substantial gender differences among the high achievers in science. 


5 Attitudes of High and Low Achievers Toward Learning 
Mathematics and Science 


To investigate whether students' attitudes were related to their achievement, we 
compared the proportions of students with specific attitudes in the groups of high 
and low achievers. 

We first established general distributions of student attitudes toward the subjects 
under investigation within each participating system (Figs. 5 and 6) While the 
response patterns were similar for both mathematics and science, we found that there 
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Fig. 5 Attitudes of grade four students toward learning mathematics. Note In Kosovo and Serbia, 
the national defined population covers 90-95% of the national target population 
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Fig. 6 Attitudes of grade four students toward learning science. Note In Kosovo and Serbia, the 
national defined population covers 90-95% of the national target population 


was considerable variation among the participants. In four of the participating Dinaric 
systems, the (vast) majority of students reported that they very much liked learning 
mathematics and science. Albania (83% for mathematics and 83% for science) and 
Kosovo (78% for mathematics and 72% for science) had the highest percentages of 
students who reported that they very much liking to learn mathematics and science. In 
Bosnia and Herzegovina, Croatia, and Serbia, more than half of the students said they 
only somewhat liked or did not like learning mathematics and science at all. Croatia 
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had the largest proportion of students reporting that they did not like learning math- 
ematics (75%), as well as the largest proportion of students who disliked learning 
science (66%). 

We then combined these attitudes toward liking to learn the subjects with student 
achievement (Figs. 7 and 8). As we anticipated, we found that students who said that 
they did not like or only somewhat liked to learn the subject were more likely to fall 
into the group of low achievers than students who said that they very much liked 
learning the subject. Conversely, those who liked the subject a lot were more likely to 
be high achievers than those who disliked or only liked learning the subject to some 
extent. For example, in Albania, 10% of the students who very much liked to learn 
mathematics did not achieve the low TIMSS benchmark (Fig. 7), while 29% of the 
students who did not like to learn mathematics so much belonged to this group of low 
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Fig.7 Distribution of students across the TIMSS international benchmarks in mathematics and their 
attitudes toward learning mathematics. Note In Kosovo and Serbia, the national defined population 
covers 90-95% of the national target population 
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Fig.8 Distribution of students across the TIMSS international benchmarks in science and their 
attitudes toward learning science. Note In Kosovo and Serbia, the national defined population covers 
90-95% of the national target population 
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achievers. We also found that the proportion of low achievers among the students, 
who responded that they only somewhat liked or did not like learning mathematics 
was quite high in five of the seven Dinaric participants. The largest proportion was in 
Kosovo (50%) and the lowest proportion was in Croatia (6%). We noted the largest 
proportion of low achievers among students who answered that they very much liked 
learning mathematics was in Kosovo (20%). 

Comparing the differences in percentages of those students who liked learning 
mathematics and those that did not like learning mathematics among the low 
achievers, we found the biggest difference in Kosovo (30%). This difference was 
smaller across the other participants in the region, with Croatia (4%) and Serbia 
(3%) reporting the smallest differences. However, all seven participants showed 
differences, indicating that positive attitudes toward mathematics were related to 
high achievement in mathematics in every system. 

Among the high achievers in mathematics, the proportion of those who liked 
learning mathematics a lot was substantially higher than the proportion of students 
that did not like learning mathematics as much (see Fig. 7). The largest percentages 
of students at or above the high benchmark who liked learning mathematics were 
found in Serbia (42%) and Croatia (41%). Out of all the participants in the Dinaric 
region, these two systems also contained the largest proportions of students who only 
somewhat or did not like learning mathematics within their groups of high achievers. 
Croatia reported the biggest difference in attitudes within the group of high achievers 
(17% more students who reported very much liking to learn mathematics); in Bosnia 
and Herzegovina, only five percent more of the high achievers reported that they very 
much liked learning mathematics. Again, all seven participants showed differences, 
indicating that positive attitudes toward mathematics were also related with high 
achievement in mathematics in every system. 

In summary, we found that, among high achievers, the proportion of students 
who reported liking to learn mathematics a lot was much higher than the proportion 
of students who reported not liking to learn mathematics as much, while, among 
low achievers, the converse was true. Positive attitudes toward learning thus tend to 
accompany high achievement in mathematics. 

In the science domain, we found that the proportion of low achievers was quite 
high among students who responded that they did not like learning science (Fig. 8). 
The largest proportions of low achievers in this group were in Kosovo (57%) and 
North Macedonia (52%), whereas the lowest proportion was in Serbia (9%). We also 
noted that the proportion of low achievers in the group of students who somewhat 
liked or did not like learning science tended to be larger than the proportion of low 
achievers among those who said that they liked learning science very much. The 
largest differences between these two groups were in North Macedonia (26%) and 
Kosovo (23%). Serbia was the only system with no reported difference between these 
groups. As observed for mathematics, these results indicate that negative attitudes 
toward learning science were related to low achievement in science across the Dinaric 
region. Students who did not like learning science much or not at all were more likely 
to be among the low achievers in science than their peers who liked learning science 
a lot. 
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Among high achievers, we found that the differences between the percentages 
of students who said that they very much liked learning science and those that said 
that they only somewhat liked or did not like learning science were small. The 
largest difference was in North Macedonia (8%), but only three participants reported 
any differences (North Macedonia, Albania, and Kosovo). Attitudes toward learning 
science do not seem to affect achievement in science among high achievers in the 
Dinaric region. 

Despite this, the higher proportion of negative attitudes toward learning among 
the lower achievers indicates that positive student attitudes toward science have a 
positive effect on science achievement. 


6 Student Well-Being and Its Relation to Achievement 


We also looked at how often grade four students reported feeling tired when they 
arrived at school (Fig. 9). Across the Dinaric region, feeling tired on arrival at school 
is an issue for a worryingly significant percentage of students. The percentage of 
students that reported feeling tired at the start of school every day or almost every 
day ranged from 13% in Albania to nearly a third of grade four students in Croatia 
(3096), North Macedonia (3196), and Bosnia and Herzegovina (32%). 

We found that, in mathematics, low achievers were significantly more likely to 
report feeling tired on arrival at school (Fig. 10). The proportion of low achievers 
was smaller and the proportion of high achievers larger in the group of students 
that answered that they never or only sometimes felt tired at the start of the school 
day than in the group of students who said they arrived tired at school every day or 
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Fig. 9 Grade four students’ reports of the frequency of feeling tired on arrival at school. Note In 
Kosovo and Serbia, the national defined population covers 90-95% of the national target population 
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Fig. 10 Distribution of students across the TIMSS international benchmarks in mathematics and 
their frequency of feeling tired on arrival at school. Note In Kosovo and Serbia, the national defined 
population covers 90-95% of the national target population 


almost every day. This difference was significant in six of the seven participating 
Dinaric systems and the differences in the group proportions were especially large 
in Albania, North Macedonia, and Kosovo. Feeling tired at the start of the school 
day thus seems to play an important role in mathematics achievement. We identified 
very similar patterns for science achievement (Fig. 11), although we note that the 
larger proportions of students who reported feeling tired in the group of science low 
achievers may be also related to the generally higher numbers in this group in some 
of the participants (e.g., Kosovo) rather than larger proportions of students feeling 
tired. 

Several factors may underlie the varying percentages of students that feel tired 
on arrival at school. The varying school starting times across the region may be one 
explanation for differences between the participating systems. Other factors may be 
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Fig. 11 Distribution of students across the TIMSS international benchmarks in science and their 
frequency of feeling tired on arrival at school. Note In Kosovo and Serbia, the national defined 
population covers 90-95% of the national target population 
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the distances that students need to travel to get to school and the means of transport 
that they have to use. For students from rural areas, the need to catch a bus to get to 
school may involve rising much earlier than their peers to reach school in time. Other 
obligations at home may also induce tiredness. A study carried out in Albania in 2017 
highlighted several reasons for absenteeism. These included, among other factors, 
the distance between school and home, particularly at the lower secondary education 
level; pressure to contribute toward the family income; additional family obligations, 
such as helping to care for other children and elders, and doing housework; and early 
marriage (Maghnouj et al., 2020). Many of these factors may also lead to tiredness 
in school, and consequently may be related to achievement. 


7 Discussion and Conclusions 


We aimed to provide an overview of the characteristics of high- and low-performing 
grade four students across the Dinaric region and to establish what could be learned 
from analyzing any observed similarities and differences. Evidence-based data can 
enable participants to formulate and implement policies and practices that support 
improvement in mathematics and science achievement. 

We found that, across the Dinaric region, there were considerable differences in 
the proportions of high- and low-achieving students in each participating system. 
The results for mathematics showed that there were more high achievers than low 
achievers in three of the participants, and the converse was true for the remainder. 
The results for science showed the same pattern. 

We analyzed a number of factors that were potentially related to the differences 
between high and low achievers in mathematics and science. A key finding was that 
there were no gender differences among low-achieving students in mathematics in 
six of the seven Dinaric participants. For science, we found that in three of the seven 
Dinaric participants the low-achieving group contained higher percentages of boys 
than girls. Among high-achieving students in mathematics, we found that, in four 
of the seven Dinaric participants, there were higher percentages of boys than girls. 
For science, the percentage of girls was higher than boys in one of the seven Dinaric 
participants. 

We also observed that, across the Dinaric region, there was a large percentage of 
students performing below the low TIMSS benchmarks, who responded that they 
did not like learning mathematics and science. However, it is also true that there 
was a considerable percentage of high achievers who said that they did not like 
learning mathematics or science. A large proportion of students felt tired on arrival 
at school; reports of feeling tired tends to be more prevalent among students belonging 
to the low achievers group. As feeling tired affects an individual's ability to listen 
attentively or work independently on specific tasks, the physical well-being of the 
student undoubtedly has an effect on their potential achievement. The relatively high 
incidence of such reports from grade four students across the Dinaric region indicates 
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that this is an issue that needs to be addressed; more research is needed to investigate 
the underlying factors. 

It is very important to reduce the number of low achievers in the student popula- 
tion. If these students do not reach minimum competencies in literacy, mathematics 
and science, this may have an impact on their future life opportunities. Govern- 
ment intervention with appropriate policies and educational practices is needed to 
avoid with future excluded citizens and a polarized society. From our perspective, all 
students should have access to similar opportunities to learn, and where conditions 
are less favorable (e.g., where students lack adequate support at home) some may 
need higher support levels in school to compensate. High achievers tend to come 
from families with high SES, where their parents invest early in creating their future 
opportunities because they are conscious of the importance of education. Efforts are 
needed to identify highly talented students with low socioeconomic status, and to 
bring or retain them in the category of high-achieving students. Education systems 
should implement special programs to ensure that these talents are not needlessly 
squandered on their journey through school; their success also affects the future 
human capital of the region, and academic success and life opportunities should not 
be determined by SES or home background. 

The existence of such large achievement differences at grade four is a critical issue 
that should be addressed by all Dinaric participants. Grade four students are at an 
age where they are consolidating the foundational skills provided by basic education 
to move into another level of education. If students have failed to reach minimum 
competencies by grade four then their future learning is endangered. Once left behind, 
the achievement gaps continue to develop, and it becomes almost impossible to 
compensate for the lack of good foundations. This may have an impact on students’ 
psychological development, potentially causing some of them to abandon school or 
complete only basic compulsory education. 

As the future learning of low achievers is in danger, it is very important to identify 
these students as early as possible and to implement measures and policies, including 
concrete teaching strategies and learning support, dedicated to better supporting their 
progress. 
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Abstract Education systems can be conceptualized as the scaffolding that supports 
the construction and development of student competences. Among other things, 
the size, location, and learning resources of schools can affect how efficient that 
system is at delivering the required support. Data from international large-scale 
assessments have indicated that the resources of rural schools may differ from those 
of urban schools; students in schools in urban and more economically developed 
environments often demonstrate higher achievement. Data from IEA’s Trends in 
International Mathematics and Science Study (TIMSS) 2019 from across the Dinaric 
region provides information on variations in the size of schools and allocation, student 
achievement, and the different kinds of scaffolding/support for learning in urban and 
rural schools. Secondary analyses of the TIMSS 2019 data for the Dinaric region, 
taking into account home and school factors, show that the types of support available 
for student learning differed between urban and rural schools. The findings suggest 
that policymakers should focus on improving the learning resources available to 
rural schools across the region, particularly in response to their lack of technological 
resources for developing students’ digital competencies. Concomitant investment 
is required for the development of teachers’ competencies for the effective use of 
such educational resources. Educators need to compensate for lack of family support 
for some students; in such situations, schools need to enhance the scaffolding for 
learning available to children. 
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1 Introduction 


In order to develop and master different competences through learning, students 
usually benefit from good support for learning: quality teaching, appropriate mate- 
rial resources for learning, and parents and teachers that nurture their academic aspi- 
rations. But does the learning support that students receive differ substantially by 
school environment? We used data collected by IEA's Trends in International Math- 
ematics and Science Study (TIMSS) 2019 to identify whether there were differences 
in learning environments across the Dinaric region, specifically between urban and 
rural schools. Seven participants from the Dinaric region took part in TIMSS 2019, 
namely Albania, Bosnia and Herzegovina, Croatia, Kosovo,! Montenegro, North 
Macedonia, and Serbia. TIMSS provides an opportunity to explore the different 
types of support (scaffolding) for student learning, enabling us to identify which 
types of support were provided in the schools in rural areas in comparison with 
schools situated in more urban locations. We hypothesized that differences among 
learning of students from rural and urban areas could also be related to students’ 
home support; we thus relate school support to differences in student achievement 
in science and mathematics between rural and urban schools, taking into account the 
support that students received, both at the home and school levels. 


1.1 Scaffolding the Learning Process 


Education systems can be conceptualized as the “scaffolding” that supports the 
construction and development of student competences (Bruner, 1983; Van de Pol 
et al., 2010; Wood et al., 1976). Here, we use the term scaffolding to encompass 
all the different ways of supporting the child in activities that they cannot perform 
independently (Wood et al., 1976) and where learning process is guided by others 
(Stone, 1998). Just as scaffolding provides support to workers in the construction of 
tall buildings and is no longer required when the building is successfully constructed, 
when a student masters a skill additional support is no longer needed and can be 
safely removed. In this sense, the term scaffolding reflects the temporary nature of 
the educational support provided as one of its important characteristics. 

The concept of scaffolding offers a new perspective in the study of classroom 
learning and suggests that learning processes can be reframed to be more attuned to 
the students’ needs. This metaphor of scaffolding can be used to identify different 
kinds of support, such as: motivating the students to work (Nedié et al., 2015; 
Seberová et al., 2020), focusing the student on certain task characteristics (Gunawar- 
dena et al., 2017; Panselinas & Komis, 2009), supportive parents (Goodall, 2020), 
using a language that the student understands, or using technical tools that facilitate 
various task-related activities (Fernández et al., 2001; Mercer, 2000). Sociocultural 


! AU references to Kosovo in this document should be understood to be in the context of United 
Security Council resolution 1244 (1999). 
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theorists point out that the dynamics of learning are largely determined by the variety 
of tools that a culture has at its disposal and the environment in which the devel- 
opment process takes place (Cole & Wertsch, 1996; Vygotsky, 1934, 2012). Some 
of these dynamics relate to cultural tools as mediators of learning, such as labora- 
tories, libraries of graphic displays, and software programs installed on electronic 
devices (such as tablets and computers), which can drastically change the learning 
process (Kozulin, 2003). In the educational context, there are numerous examples of 
support, such as asking students constructive questions, using tools that can lead to 
more meaningful learning, and teaching students how to communicate their thoughts 
(Fernández et al., 2001; Mercer, 2000; Mercer & Littleton, 2007; Radišić & Jošić, 
2015). 

The extent to which the school emphasizes the importance of student academic 
success is one of the school factors underpinning student achievement (Goddard et al., 
2000; Hoy et al., 1991). Setting high, but achievable goals in terms of student achieve- 
ment leads to the establishment of an orderly and effective learning environment and 
motivates students to work and achieve better results. 

The size, location, and learning resources of the school may affect the efficiency 
of the school as a system. Equipped with all the necessary elements (library, gymna- 
sium, and laboratory), smaller schools are usually more effective, providing a safe 
and intimate learning environment (Klonsky, 2002; Wasley et al., 2000). However, 
research into the importance of school resources for achievement has proved incon- 
sistent; some studies found that these resources are not critical to student success 
(Hanushek, 1997), while others reported that the amount of money spent by a school 
per student was a strong predictor of achievement (Hedges et al., 1994). For the 
efficient use of information technologies, employee training is certainly necessary 
(Fraillon et al., 2020; Laffey et al., 2003). 

All these types of scaffolds can support different subjects, such as science or 
mathematics (Dawes, 2008; Mercer et al., 2004). The scaffolding concept can also 
be useful as an analytical tool to help gain a greater understanding of teaching and 
learning in schools at different levels of urbanization. 


12 Urban and Rural Schools in the Dinaric Region 


Defining the terms urban and rural is a rather challenging task because there does 
not seem to be one unambiguous answer that permits a universally understood defi- 
nition to be assigned. Historically, the term rural meant something that was “out- 
side the city walls.” From the economic aspect, rural territory is used to produce, 
above all, food, while sociologists might characterize the rural environment as more 
technologically and culturally backward in development terms than an urban envi- 
ronment. Today, various criteria are used to analyze rural and urban concerns, such 
as demographic criteria, the amount and structure of the population’s income, loca- 
tion criteria and measures of basic activity of inhabitants in a certain territory. Rural 
areas are thus characterized as sparsely populated places, places where people have 
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lower incomes, and areas that have a different purposes from urban areas, primarily 
dominated by agricultural land that often defines the activities and professions of 
inhabitants. TIMSS 2019 created general international definitions of rural and urban 
that were based on the number of inhabitants in the region in which the school was 
located, but subcategories of urbanization were more precisely defined by popula- 
tion sizes for cities, towns, and villages in each education system (Mullis & Martin, 
2017). 

Students in schools in urban and more economically developed environments 
often demonstrate higher achievement. This may berelated to the availability of better 
teaching staff, better local community resources, or higher socioeconomic status 
(SES) of families in many urban classrooms (Darling-Hammond, 1996; Erberber, 
2009; Hooper et al., 2013; Mohammadpour & Abdul Ghafar, 2014; Piyaman et al., 
2017). The relations between achievement and learning environment in urban and 
rural schools are also reflected in the education systems across the Dinaric region. 
For instance, in Serbia, results from the national test in mathematics and Serbian 
language in 2004 showed grade three students from urban areas achieved higher 
scores than their peers in rural schools (Baucal et al., 2007). The results also showed 
that differences in achievement between students from rural and urban schools could 
be mostly explained by their social background and different preparation for starting 
school, and only to a smaller extent by variance in the quality of mathematics teaching 
in rural and urban schools. Serbia and Albania showed similar urban-rural gaps 
in reading scores on the Programme for International Student Assessment 2018 
(PISA) of about 45 points (OECD [Organisation for Economic Cooperation and 
Development], 2019). 

According to a report by the United Nations Children's Fund in Bosnia and Herze- 
govina (UNICEF, 2020) into the situation of children in Bosnia and Herzegovina, 
about 60% of young people under the age of 18 lived in rural areas, but children in 
rural areas made up only 0.5% of the total number of children attending preschool 
education. Data on student achievement for rural areas are inconsistent. On the one 
hand, results from the 2011 Labour Force Survey (Somun-Krupalija, 2011) for the 
whole of Bosnia and Herzegovina and their Survey of Rural Households in 2012 
(which included people living outside urban settlements; Goss, 2012) showed that 
the population of Bosnia and Herzegovina that lived in the countryside tended to be 
less educated than its urban inhabitants. Rural residents received, on average, two 
years less education than those in urban areas, regardless of gender (Goss, 2012). 
On the other hand, the reports of United Nation Development Programme (UNDP) 
about rural development (UNDP, 2013) as well as the Multiple Indicator Cluster 
Survey report (The Agency for Statistics of Bosnia and Herzegovina et al., 2013) 
found that the education in rural areas is improving. Villagers were becoming less 
disadvantaged in terms of formal education because most manage to send children 
in school despite the distance and the total number of children in Bosnia and Herze- 
govina attending high school has increased significantly, from 74% in 2006 to 92% 
in 2011/12 (UNDP, 2013). We caution that differences in reported statistics arise 
from their different focus, but also from the methodology of defining rural areas by 
cantons in Bosnia and Herzegovina. 
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Regardless of the interpretations of the fairness of education in different envi- 
ronments, all reports agree that the extremely complex administrative organization 
of the education system in the region makes it difficult to harmonize regulations 
on education and service delivery, especially in rural areas. Variance in preschool 
education across the different regions was also noted in North Macedonia, where 
about 32% of 0—5 year olds attended preschool education in the East and Pelagonia 
regions, but only eight percent in the Northeast region (World Bank, 2019). Data for 
North Macedonia shows that the total enrollment of children in education in rural 
areas was lower than in urban areas (Eftimoski, 2006). 

The socioeconomic background of students from different regions is generally 
related to levels of education across all the Dinaric systems that participated in 
TIMSS. The UNDP report for 2019 (Conceicáo, 2019) showed that some territories 
face regional differences in the level of development, as measured by the UNDP’s 
human development index, which includes an education index (EI) as one of the indi- 
cators of human development. The EI was composed of the average adult’s number 
of years of schooling and the expected number of years of schooling for children 
in the region, each receiving a 50% weighting. Capital cities, like Tirana in Albania 
and Belgrade in Serbia, had higher Els than other areas, showing that there were 
clear differences between urban and rural regions (Baucal et al., 2007; Vujni¢, 2014; 
World Bank, 2019). A World Bank (2019) report on the effects of urbanization in 
Albania, Bosnia and Herzegovina, Croatia, Kosovo, North Macedonia, Montenegro, 
and Serbia, using data on educational inputs, suggested that regions with lower EIs 
were often well resourced. For example, relative to their populations, regions like 
Diber and Kukés in Albania had more schools and teachers and smaller classes than 
many more affluent regions. This partially reflects an explicit effort to ensure access 
to education regardless of where people live, but also reveals the demographic decline 
in rural areas; schools remain, but declining populations mean fewer students attend 
(World Bank, 2019). 

A search of the literature revealed that education systems in the Dinaric region 
recognize dichotomous classification (urban/rural) and generally use the number of 
inhabitants as the classification criteria, so this was a logical choice for our research 
(see, e.g., Gajić, 2015; Milanović et al., 2010; Miljevic-Ridicki et al., 2011; Rajovic & 
Bulatovic 2015; Somun-Krupalija, 2011). Data collected by TIMSS 2019 provided 
important information about education in schools at different urbanization levels. 
The TIMSS 2019 contextual framework classified school location using five cate- 
gories: urban, suburban, medium-sized city or large town, small town or village, and 
remote rural; each category was contextualized using relevant national definitions of 
population size (Mullis & Martin, 2017). Based on previous research into an earlier 
cycle of TIMSS in 2015 (see Boulifa & Kaaouachi, 2015; Webster & Fisher, 2000), 
for our analyses we merged the first three categories into one signifying urban areas 
and the other two into one signifying rural areas. We used this simplified type of 
classification because this enables us to obtain comparable data on schools from 
different education systems with nationally defined numeric criteria for urban loca- 
tions. More detailed information about the Dinaric education systems can be found 
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in chapter “Introduction to Dinaric Perspectives on TIMSS 2019” and in the TIMSS 
2019 encyclopedia (Kelly et al., 2020). 


13 Scope of the Chapter 


Policymakers across the Dinaric region are Interested in learning whether all students 
have equal access to education and whether all children have equal learning support 
(Boljka et al., 2018; European Commission, 2014; Krstevska & Trenceva, 2016; 
UNDP, 2013; OECD, 2019). The dilemma that immediately arises when considering 
schools in rural areas is their cost-effectiveness over educational effectiveness. In 
particular, the enduring question is whether schools in rural areas should be supported 
by national funding or whether their funding should be reduced by placing students 
in schools in more urban areas, while subsidizing the additional travel costs to enable 
students from more distant locations to attend. 
Our work was guided by three key research questions: 


(1) Do schools from different urbanization levels (urban and rural) differ in other 
defined ways (for example, school size)? 

(2) Do urban and rural schools differ in terms of their student achievement? 

(3) Do urban and rural areas differ in terms of the types of scaffolding available 
to support student learning in schools? 


2 Dataand Methods 


We analyzed TIMSS 2019 data from across the Dinaric region, collected from grade 
four students, their parents, and their school principals (for more information, see 
TIMSS & PIRLS International Study Center, 2018). All the variables that we used 
in our analyses are available in the TIMSS international database (Table 1). We used 
TIMSS data for mathematics and science achievement at grade four to document the 
achievement gaps between schools in urban and rural areas and as outcome variables 
in regression models. We calculated percentages, means and regression models, and 
used t-test statistics to determine statistical group differences (for a detailed descrip- 
tion of the data sources, methods, and procedures used in our analyses, please see 
Sect. 5). 

As part of the TIMSS school questionnaire, principals were asked to assess the 
level of urbanization of the area in which their school was located. As already 
mentioned, we derived the categories of urban and rural that we used in our anal- 
yses from the five internationally defined categories (see Table S.17 in the supple- 
mentary materials available for download at www.iea.nl/publications/RfEVol13 for 
disaggregated results). 

To investigate the reasons for the differences in the achievement of students in 
relation to the type of school, we analyzed a variety of home and school support for 
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students, and the availability of material resources at schools, such as laboratories, 
libraries, and technology equipment. To assess the level of home support, we used 
principals’ reports of student readiness for schooling and the TIMSS scale home 
resources for learning, and to assess the level of school support, we analyzed school 
emphasis on academic success and school material resources. 

Students differ in the degree to which they know how to read, write and count 
when entering grade one. To indirectly measure how prepared students are when 
starting school, TIMSS asks their principals to estimate how much students know on 
entering school. The TIMSS scale “Schools where students enter the primary grades 
with literacy and numeracy skills” combines principals’ responses to several items on 
the school questionnaire to provide a measure of students’ readiness for schooling. 
Another factor that may mediate the effect of urbanization could be differences in 
parental support for schooling. The TIMSS “Home resources for learning” (HRL) 
scale attempts to measure this by combining student responses from the student ques- 
tionnaire and the responses of their parents/guardians from the early years question- 
naire to statements related to resources that are available in the home learning envi- 
ronment. To assess school scaffolding, we focused on two sets of data in the TIMSS 
database. The first was the TIMSS scale “School emphasis on academic success,” 
(SEAS) which combines principals’ responses to a number of items designed to 
measure their perceptions of the school community’s focus on student achievement. 
The second was the principals’ assessments of school material resources, such as 
availability of technology equipment, laboratories, and libraries. As an additional 
assessment of the material resources that may be available to students in schools, 
we also investigated the student: computer ratio in the grade four classrooms, the 
availability of online learning management systems (OLMSs), and access to digital 
resources in school. 

To better understand how all these factors were related to student achievement, 
we undertook regression analyses to predict mathematics and science achievement 
based on the urbanization level of schools, and the home resources for learning and 
school emphasis on academic success scales. 


3 Results 


3.1 Allocation and Size of Schools in Urban and Rural Areas 


We found that the percentages of urban and rural schools and the percentages of 
students in these schools varied across the Dinaric region (Table 2). In Albania, 
Croatia, Kosovo, and Serbia, about a third of schools were located in urban areas. 
In Bosnia and Herzegovina and North Macedonia, slightly less than half of schools 
were located in urban areas, while, in Montenegro, more than half of schools were 
located in urban areas. Regarding the percentages of students in schools by location, 
Montenegro reported that around 85% of students were enrolled in schools in urban 
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Table 2 Percentage of schools and students in schools by urbanization of location area of the 
school 


Education system Schools in Students in Schools in Students in 
urban areas urban areas (%) | rural areas (%) | rural areas (%) 
(%) 
Albania ER (3.3) | 37 (2.9) 
Bosnia & Herzegovina | 44 (3.0) | 40 (3.2) 
Croatia 36 (5.0) | 43 (3.3) 
Kosovo? 32 (4.1) | 43 (3.2) 
Montenegro 55 (2.9) 15 (0.5) 
North Macedonia 44 (2.6) | 36 (3.3) 
Serbia* 36 (5.7) | 32 (3.0) 


Notes Standard errors appear in parentheses 
*National defined population covers 90-95% of the national target population 


areas, while, in most of the other participating education systems, at least a third of 
students attended schools in rural areas. 

There were large differences in the average numbers of enrolled students, both 
in total and at grade four, between urban and rural schools in all participating 
systems (Fig. 1). Schools in urban areas had significantly more students than those 
in rural areas. However, these size differences vary across the Dinaric region. In 
Kosovo, urban schools were, on average, three times as large as rural schools, and, in 
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Fig. 1 School size (mean total enrollment) of urban and rural schools 
Note In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population 
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Montenegro, urban schools were about twice as large as rural schools; Croatia showed 
the smallest difference in school size by area (for in depth comparisons of average 
school size indicators by urbanization level, see Table S.18 in the supplementary 
materials available for download at www.iea.nl/publications/RfE Vol 13). 

We used the variable school composition by socioeconomic background to cate- 
gorize schools into three groups according to the number of enrolled students with 
different socioeconomic background. Across most of the Dinaric region, there were 
more students in more affluent schools in urban areas; the exception was Kosovo 
(Fig. 2). In North Macedonia and Croatia, more than 60% of students in urban 
schools attended more affluent schools. Conversely, rural schools were more likely 
to be more disadvantaged than urban schools. There was more variation within the 
group of school that were categorized as neither more affluent nor more disadvan- 
taged; it is worth noting that, in North Macedonia, just five percent of students in 
urban areas were in this intermediate category of schools, suggesting severe social 
segregation existed in these urban areas. 

As schools from different regions differ in terms of the socioeconomic background 
of their student bodies, teachers and schools in different regions experience different 
work conditions. Some teachers work in classes containing many students from 
economically affluent homes, while others work in classes where many students 
come from economically disadvantaged homes. 


3.2 Students Achievement in Urban and Rural Areas 


As our focus was on identifying the differences between urban and rural areas rather 
than across the region, we limited comparisons to the differences between the mean 
mathematics and science achievement among students in urban and rural schools, 
which we term the urban-rural achievement gap (Fig. 3). 

First, we must note that there were statistically significant differences in student 
achievement depending on the urbanization of the student's school area across the 
whole of the Dinaric region. Students in urban schools had consistently higher mean 
achievement in mathematics and science than their peers in rural schools. The only 
exception was in Montenegro, where there was no difference in the mean achievement 
of students in schools of different urbanization levels in mathematics, although there 
was a difference for science. The biggest urban-rural achievement gaps were in 
Serbia and North Macedonia, in both mathematics and science; in both, the mean 
achievement of students in urban schools was 36—45 points higher than students 
attending rural schools. The achievement gaps in Albania and Kosovo were much 
smaller for both subjects, 18 and 25 points, respectively. Bosnia and Herzegovina, 
Croatia, and Montenegro formed a third group where mean achievement gaps for 
both subjects were less than 15 points, although still significant. 
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Fig. 2 Percentage of students in urban and rural schools by socioeconomic background of the 
student body 

Notes Percentages add up to 100% for urbanization levels, allowing direct comparisons within and 
across economic groups. In Kosovo and Serbia, the national defined population 90-95% of the 
national target population. In Kosovo, data are available for >50% but <70% of students. (a) more 
affluent; (b) neither more affluent nor more disadvantaged; and (c) more disadvantaged 
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Fig. 3 Achievement difference in mathematics and science between urban and rural areas 

Notes Positive values mean scores in urban areas were larger. Differences were statistically signifi- 
cant (p « 0.05) in all but in Montenegro for mathematics. In Kosovo and Serbia, the national defined 
population covers 90-95% of the national target population 


3.3 Urbanization and Different Kinds of Learning Supports 
for Students 


Home Scaffolding 


We used a TIMSS scale based on principals' estimates of the percentage of students 
in the school who possessed a range of literacy and numeracy skills when beginning 
primary school (the schools where students enter the primary grades with literacy and 
numeracy skills scale) as a measure of students' readiness for schooling. Comparing 
data for rural and urban schools in the seven Dinaric participants, we found that, in 
general, there were few statistically significant differences in readiness for school 
across the region. Statistically significant differences were only found in Montenegro 
and Serbia, where the data showed that, according to the principals, more students 
entering school with literacy and numeracy skills in urban schools than in rural 
schools; the differences were more pronounced in Serbia. 

As a further indicator of the effects of urbanization, we analyzed differences in 
mean scores on the HRL scale; these were statistically significant for all partic- 
ipants (Fig. 4). This means that students in rural schools had, on average, fewer 
home resources (such as books, desks, their own room, and internet access) and less 
educated parents with lower occupational levels than their peers in urban schools. The 
differences the mean scores were largest in Albania, Serbia, and North Macedonia. 


School Scaffolding 


In general, according to principals' reports, schools across the Dinaric region placed a 
high emphasis on academic success (Fig. 5). However, there were statistically signif- 
icant differences between schools in urban and rural areas in Croatia, Montenegro, 
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Fig. 4 Mean scores on the TIMSS scale home resources for learning for groups of urban and rural 
schools 

Notes In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
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and Serbia. In all three of these systems, principals of urban schools tended to 
report higher levels of emphasis on academic success in their school community 
than principals of rural schools. 

Our investigation of the additional material resources that may be available to 
students in schools, showed that the difference between percentages of students 
enrolled in urban and rural schools where an OLMS was available was only signifi- 
cant in Albania, where 19% of students in urban schools and only seven percent of 
students from rural schools had access to OLMSs. For the remainder of the Dinaric 
participants, urban and rural schools this indicator did not differ, nonetheless, the 
total percentage of schools using an OLMS varied across the Dinaric region (see 
Table S.19 in the supplementary materials available for download at www.iea.nl/ 
publications/RfEVol13). In Albania and Kosovo, there were only low percentages of 
students in schools that had an OLMS, while Serbia, North Macedonia, and Croatia 
reported higher percentages of students were in school that had access to an OLMS. 

Our assessment of the availability of technology for students in school and whether 
this availability differed in relation to school urbanization indicated that, overall, 
schools in the region were not well equipped with computers and there was consid- 
erable variation regarding availability to students (Fig. 6; see Table S.19 and S.20 
in the supplementary materials available for download at www.iea.nl/publications/ 
RfEVol13). While in most of the Dinaric education systems, there was one computer 
for approximately every 10 students, the availability of computers was noticeably 
lower in urban schools in Kosovo. In North Macedonia, a computer was shared 
between four students in urban schools and between two students in rural schools. 
The student:computer ratio was lower in urban than rural schools in Albania, Croatia, 
Kosovo, and North Macedonia. Kosovo had more computers per student in rural 
schools than in urban schools, suggesting that they had invested in providing this 
type of additional school scaffolding to rural communities. Access to digital resources 
was also inconsistent for students in Albania, Kosovo, and Montenegro, where the 
percentage of students in schools with access to digital resources was higher in urban 
than in rural areas. 


Laboratories 


Overall, data for schools that had a laboratory and provided assistance for conducting 
science experiments showed that there were few significant differences (Table 3). 
Students in urban schools in Albania had greater access to laboratories (2696) and 
were provided more assistance with conducting science experiments (2196) then 
their peers in rural schools (5% and 6%, respectively). Conversely, in Montenegro, 
a larger percentage of students in rural schools had access to laboratories (2896) 
and assistance with conducting science experiments (50%) than their peers in urban 
schools (2096 and 23%, respectively). While Serbia reported no significant differ- 
ences between urban and rural schools regarding the availability of laboratories, there 
was a significant difference in the availability of assistance for conducting science 
experiments. In urban schools 39% of students received this kind of support during 
teaching, while only 19% of students in rural schools had this support. 
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Fig. 6 Assessment of technology resources in urban/rural schools: a the average student:computer 
ratio in grade four classrooms; b percentage of students that had access to digital resources in school; 
and c percentage of students in schools that had access to an online learning management system 
Notes In Kosovo and Serbia, the national defined population covers 90-95% of the national target 
population. *differences were statistically significant (p < 0.05) 
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Table 3 Percentages of students in schools with a science laboratory and in schools that provide 
assistance for conducting science experiments 


Education system Percentage of students in schools | Percentage of students in schools 
where a science laboratory is where assistance for conducting 
available science experiments is available 
Urban Rural Difference | Urban Rural Difference 

Albania 26 (3.6) |5(2.7) |21(4.9) 21(4.2) |6 (2.5) |15(4.8) 

Bosnia & Herzegovina | 19 (4.8) | 7 (3.2) 12 (6.6) 24 (4.8) |28 (5.7) | —5 (7.8) 

Croatia 6(2.7 |6(3.6) | 0 (4.5) 15 (3.7) | 15 (4.8) | 0 (6.0) 

Kosovo* 24 (6.2) | 14 (4.5) | 10 (7.9) 14 (4.5) |8(3.7) |6(5.7) 

Montenegro 20 (0.3) | 28 (1.2) | —8 (1.2) 23 (0.4) |50(2.1) | —27 (2.1) 

North Macedonia 4(2.2) | 9 (3.9) —5 (4.3) 38 (5.1) | 25 (6.4) | 14 (8.4) 

Serbia? 11 (2.9) |10(3.6 | 1 (4.4) 39 (5.4) | 21 (5.8) | 18 (7.7) 


Notes Statistically significant (p < 0.05) differences are shown in bold. Standard errors appear in 
parentheses 
*National defined population covers 90-95% of the national target population 


Libraries 


When comparing the availability of school libraries in urban and rural schools, we 
found only Kosovo and Northern Macedonia showed significant differences (Table 
4). In Kosovo, 90% of students in urban schools had access to a school library, while 
this percentage was lower for students in rural schools (73%). A slightly smaller 
percentage difference occurred in North Macedonia, where all students in urban 
schools had libraries compared to only 89% of students in rural schools. 

Some schools reported having large libraries, containing more than 2000 books; 
for this factor the differences were only significant in Albania and North Macedonia. 
In Albania, 28% of students attending urban schools had access to large libraries, 
while only one percent of students in rural schools had similar resources. Data for 
North Macedonia indicated that large 76% of students in urban schools had access 
to large school libraries, compared to only 51% of students in rural areas. 

Classroom libraries were available for small percentages of students in all seven 
TIMSS participants, but the urban-rural difference was only significant in Kosovo 
and Montenegro. In urban schools, 16% of students in Kosovo and 18% of students 
in Montenegro had classroom libraries; the comparable figures for rural schools were 
three percent of students in Kosovo and 11% of students in Montenegro. 


3.4 Student Achievement in Urban and Rural Areas 
Regarding the Type of Scaffolding 


To better understand how all these factors were related to student achievement, we 
undertook regression analyses to predict mathematics and science achievement from 
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Table 5 Amount of variance in student mathematics achievement by school location, the home 
resources for learning scale (HRL) and the school emphasis on academic success scale (SEAS) 


Education Number | Variance | Standardized regression coefficients 
2 

system of (RY) Urbanization SEAS HLR 

students | explained 

(n) by model 
Albania 3986 0.17 —0.03 | (0.05) 0.13 | (0.04) | 0.38 | (0.03) 
Bosnia and 5260 0.12 0.00 | (0.03) —0.02 | (0.03) | 0.35 | (0.02) 
Herzegovina 
Croatia 3684 0.13 —0.01 (0.03) 0.01 | (0.03) | 0.36 | (0.03) 
Kosovo* 4256 0.09 0.04  |(0.04) 0.09 | (0.03) | 0.27 | (0.02) 
Montenegro | 4325 0.13 —0.05 | (0.02) —0.01 | (0.02) | 0.37 | (0.01) 
North 2685 0.23 0.01 (0.05) 0.06 (005) 0.45 | (0.02) 
Macedonia 
Serbia? 4184 0.27 0.06 | (0.03) 0.05 | (0.03) | 0.49 | (0.02) 


Notes To assess the urbanization level of schools, we coded rural schools as 0 and urban as 1. R2 = 
the proportion of variance in the outcome variable that is explained by the set of predictor variables. 
Statistically significant (p < 0.05) regression coefficients are shown in bold. Standard errors appear 
in parentheses 

*National defined population covers 90-95% of the national target population 


the urbanization level of schools, home resources for learning scale scores, and the 
school emphasis on academic success scale scores (where higher values mean more 
home resources were available and the school placed greater emphasis on academic 
success) (Tables 5 and 6). 

The three predictors together explained between nine and 27% of variance in 
achievement across the Dinaric region, which is a remarkable amount given the many 
factors that potentially affect student achievement. Home resources for learning was 
a significant predictor for all participants after controlling for urbanization and school 
emphasis on academic success. While school emphasis on academic success was a 
significant predictor of grade four students” mathematics and science achievement 
in Kosovo and Albania, the urbanization level of the school was only significant for 
mathematics achievement and only in Montenegro and Serbia. Nevertheless, all these 
significant coefficients were rather low, and, after controlling for school emphasis 
on academic success and home resources for learning, the difference in achievement 
scores between urban and rural schools disappeared. In other words, differences in 
student achievement between urban and rural schools seem to be largely determined 
by the students’ backgrounds, and these may vary considerably according to the 
urbanization of the area surrounding the school. 
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Table6 Amount of variance in student science achievement by school location, the home resources 
for learning scale (HRL) and the school emphasis on academic success scale (SEAS) 


Education Number | Variance | Standardized regression coefficients: 
2 

system of RI) Urbanization SEAS HLR 

students | explained 

(n) by model 
Albania 3986 0.16 —0.01 (0.05) 0.14 (004) |0.36 | (0.03) 
Bosnia and 5260 0.10 0.01 (0.03) —0.01 | (0.03) | 0.31 | (0.02) 
Herzegovina 
Croatia 3684 0.13 —0.01 (0.03) 0.02 | (0.03) | 0.37 | (0.02) 
Kosovo* 4256 0.10 0.07  |(0.04) 0.11 | (0.03) | 0.26 | (0.03) 
Montenegro | 4325 0.14 —0.03 (0.02) —0.01 | (0.02) | 0.38 | (0.02) 
North 2685 0.25 0.04  |(0.05) 0.08 (005) 046 | (0.03) 
Macedonia 
Serbia? 4184 0.27 0.06 | (0.03) 0.05 | (0.03) | 0.49 | (0.02) 


Notes To assess the urbanization level of schools, we coded rural schools as 0 and urban as 1. 
R? = the proportion of variance in the outcome variable that is explained by the set of predictor 
variables. Statistically significant (p < 0.05) coefficients are shown in bold. Standard errors appear 
in parentheses 

*National defined population covers 90-95% of the national target population 


4 Discussion 


International studies have noted differences in educational achievement between 
urban and rural schools, generally in favor of the urban schools (see e.g., Moham- 
madpour & Abdul Ghafar, 2014; Piyaman et al., 2017; Wasley et al., 2000). We aimed 
to identify variables which could enhance understanding of differences between 
urban and rural schools. We analyzed any differences and similarities between these 
two groups of schools in Dinaric region and analyzed the type and quality of the 
scaffolding for learning that students received from both home and school. 

Our comparisons confirmed that urban and rural schools tended to have different 
demographic structures. In all seven participating systems, there were more students 
in total and more grade four students enrolled in urban schools than rural schools. Not 
only were there fewer students enrolled in rural schools, but these schools are also 
smaller. We also analyzed principals' reports of the percentage of enrolled students 
that came from homes with different socioeconomic situations. The data revealed 
that, in general, the percentages of students enrolled in the schools categorized as 
more affluent tended to be higher in urban areas than in rural areas. 

Students' TIMSS achievement scores can usually be attributed to a combination 
of factors, including factors related their family background, and other school- or 
teacher-related factors. We wanted to establish whether the support that comes with 
these factors differed between urban and rural areas. At the family level, we assessed 
variables related to student readiness for schooling and home resources for learning. 
In general, scaffolding in learning measured by scores on the schools where students 
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enter the primary grades with literacy and numeracy skills scale did not appear to 
differ substantially between urban and rural schools; students in five out of the seven 
participating systems tended to achieve similar mean scores in both urban and rural 
schools. However, in Montenegro and Serbia, principals of students in urban areas 
tended to report that their children started school more prepared in comparison to 
the reports from principals of rural schools. While the analyses for most Dinaric 
participants showed no differences in principals” perceptions of the readiness of 
students for schooling between urban and rural schools, there were differences in 
home resources for learning. In all participating seven Dinaric systems, students 
from urban areas had better access to resources such as books in the home and home 
study supports, and tended to have parents with higher educational and occupational 
levels. 

At the school level, we analyzed the factor school emphasis on academic success 
and the material resources for learning available in the school. School emphasis 
on academic success differed between urban and rural schools in only three of the 
education systems, while in other four there was no difference between urban and 
rural schools. In Croatia, Montenegro, and Serbia, school emphasis on academic 
success was lower in rural areas than in urban areas. This led us to investigate school 
education policies in these systems to discover the reasons underlying such differ- 
ences. We also examined the material resources that were available to students in 
schools. In general, in most of the participating systems, urban and rural schools 
has similar levels of access to technology, libraries, and laboratories. There were a 
few exceptions to this, which may provide important lessons for improving learning 
environment in rural schools. For instance, students in urban schools in Albania had 
greater access to laboratories and to assistance in conducting science experiments 
than students in rural schools. In Kosovo and North Macedonia, the percentage of 
rural students that had access to libraries was significantly lower than percentage 
of students that such critical support in urban area. These findings provide a good 
basis for policymakers to discuss when considering the topic of rural schools. All 
Dinaric education systems are advised to devote more attention to equipping rural 
schools with the requisite technologies and resources to compensate students for the 
reduction in educational opportunities created by lack of such resources in the home. 

Our analyses of the TIMSS 2019 data confirms that an achievement gap between 
urban and rural areas exists in all seven Dinaric systems. The achievement gap was 
substantial in Serbia and Kosovo (up to 40 points), both for science and math- 
ematics achievement. Our regression analyses showed that, after controlling for 
school emphasis on academic success and home resources for learning, the differ- 
ence in achievement scores between urban and rural schools disappeared. Note that, 
in our analyses, we simplified the distinction between these two groups of schools 
(urban and rural), but there may be additional differences between densely populated 
metropolitan capital cities and other districts.’ 


? National achievement testing (Baucal et al., 2007) and analysis of the human development index 
results (Vujnié, 2014) showed, for example, that treating the center of Belgrade (Serbia) as a separate 
region returned different perceptives and prediction models for achievement. 
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Analyses of the TIMSS 2019 results provide valuable evidence-based data for 
both policymakers and those professionally engaged in topics related to urbaniza- 
tion and education. We have identified several key points that are significant across 
the Dinaric region: (1) there needs to be increased awareness of the difference in 
achievement in students from different locations; (2) rural areas often lack resources 
for learning in the home, putting students from these areas at a disadvantage; and 
(3) all schools require the requisite technologies and equipment to support their 
students, while the demands may differ by location. The overarching message is 
that there are students who experience very different learning conditions created by 
location, and their achievement may thus differ. We suggest that families from rural 
areas require significant support in order to minimize the differences in learning 
outcomes among students. Schools may need additional support to provide the labo- 
ratory equipment, materials, computers, and software that can help in better devel- 
oping students’ competencies. Of course, the availability of school resources does 
not automatically mean that they will be used in the classroom; teachers also need to 
have the knowledge and skills to use the available resources successfully. Therefore, 
resources need to be accompanied by investment in the professional development of 
teachers’ competencies so that educational resources are used most effectively. As 
well as equipping households with computer equipment and supplying books, fami- 
lies will need additional support to make optimal use of the materials. The learning 
process in rural schools needs to be constantly reviewed, to broaden understanding 
of the factors affecting student achievement. In future analyses, it would be inter- 
esting to compare this Dinaric data with similar research efforts in other European 
countries, or even a more global TIMSS context. 
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